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ABSTRACT 


All too often, system requirements are not funded appropriately due to failure to 
incorporate Human Systems Integration (HSI) costs into the analysis of 
alternatives (AoA) in the acquisition process. Failure to incorporate HSI costs 
results in cost overruns and deficiencies in performance that have a significant 
impact on life-cycle costs. For a cost estimate to be accurate, all HSI domains 
(e.g., manpower, personnel, training, human factors) must be included. A 
thorough cost estimation process for an AoA should identify, justify, and estimate 
all HSI-specific activities. The current cost estimation processes are lacking 
these HSI-specific activities. 

This thesis describes the initial development and evaluation of an HSI cost 
estimation process for use during the AoA. A database was created to capture all 
HSI-related activities during each phase of the acquisition process. Forty 
subject-matter experts (SMEs) reviewed and provided feedback on the database 
and on a list of HSI cost drivers. They also estimated the percent effort required 
for each HSI activity. SMEs concurred with 80% of the database and made valid 
recommendations for improving the remaining 20%. This initial framework 
represents a first step in providing a process for estimating the cost of HSI 
activities. 
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EXECUTIVE SUMMARY 


All too often, system requirements are not funded appropriately due to failure to 
incorporate Human Systems Integration (HSI) costs into the analysis of 
alternatives (AoA) in the acquisition process. Failure to incorporate HSI costs 
results in cost overruns and deficiencies in performance that have a significant 
impact on life-cycle costs. For a cost estimate to be accurate, all HSI domains 
(e.g., manpower, personnel, training, human factors) must be included. A 
thorough cost estimation process for an AoA should identify, justify, and estimate 
all HSI-specific activities. 

This thesis describes the initial development and evaluation of the AoA 
HSI cost estimation process. The purpose of the HSI cost estimation process is 
to support budget development and justification over the life cycle of the system. 
The process presented in this thesis complements the management structures 
within the Integrated Defense Acquisition, Technology, and Logistics (IDAT&L) 
Life Cycle Management. By employing this process, HSI practitioners and other 
acquisition professionals will have access to information to support decision 
making throughout the life cycle of the system and to reduce total life cycle costs. 
This process was developed in conjunction with the HSI practitioners from the 
United States Air Force and Navy; however, it can be applied by all services and 
agencies within the Department of Defense and the federal government. 

Two research efforts were carried out to evaluate the utility of the HSI cost 
estimation process. The first effort reviewed existing HSI instructions and 
guidance, and cost estimation methodologies. Information obtained from this 
research was used to design a database of specific HSI activities that require 
cost estimation during to the AoA. The second effort evaluated the content of the 
database for relevance and accuracy. The database was given to 83 U.S. Air 
Force and Navy subject-matter experts (SMEs) in the fields of HSI, defense 
acquisition and cost estimation; 40 responses were received. SMEs were 



generally satisfied with the database. They concurred with 80% of the database 
and made valid recommendations for improving the remaining 20%. The SMEs 
agreed that, once fully developed, the database would serve as a basis for a 
valuable HSI-specific cost estimation process that could support the HSI 
community during the AoA. 

In order to improve the database, several SMEs recommended identifying 
the most important HSI activities in each of the oversight documents, including 
the following: 

• Review Acquisition Strategy; 

• Acquisition Plan; 

• Source Selection Plan; 

• Request For Proposal; 

• Final Product Baseline; 

• Critical Safety List Items; 

• Programmatic Environment, Safety and Occupational Health 
Evaluation (PESHE); and, 

• TEMP. 

In addition, steps should be taken to clarify HSI activities in the Acquisition 
Management, Acquisition Engineering and Support/Sustainability activity tiers. 
Other recommendations included reorganizing the database in terms of work 
breakdown structure cost elements, identifying the benefits/impacts and risk/cost 
relationships of HSI activities. 

Further development and evaluation of the database is required beyond 
what was completed for this thesis. These refinements include comparing HSI 
cost methods and developing the HSI variables that could be included in a cost 
model; determining the domain tradeoffs that must occur to achieve accurate 
cost estimates; and, using a system of interest in a case study to validate the 


XX 




database. The initial framework developed in this thesis represents a first step in 
providing a process for estimating the cost of HSI activities in the acquisition 
process. 
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I. INTRODUCTION 


A. BACKGROUND 

Department of Defense (DoD) Instruction 5000.02 has directed Program 
Managers to take steps to include HSI considerations into their planning, 
development, design and execution of programs. This is necessary in order to 
minimize total ownership costs and also to optimize total system performance as 
early as possible in the acquisition process (Department of Defense, 2008c). In 
order to accomplish these tasks, line item costs from approved legacy system 
Work Breakdown Structures (WBS) are treated as baselines for component 
equipment (hardware, software, etc.) as well as manpower, then incorporated 
into the estimates for funded requirements. A formal cost estimation process for 
the components is also completed. All too often, requirements are not funded 
appropriately due to failure to incorporate HSI costs into the Analysis of 
Alternative process for various military system capabilities. These failures 
produce cost overruns and deficiencies in performance resulting in overall 
significant impact on the life-cycle cost. The development of a standardized HSI 
cost estimation model is needed. Such an HSI cost estimation model would fulfill 
the requirements of the system and define the system in terms of relevant 
activities specified by cost estimation relationships. The benefits and risks could 
reduce the impact on life-cycle cost. This model would provide program 
managers with an enhanced decision-making capability. 

To provide appropriate measures, the methodologies need to encompass 

size or complexity drivers and life cycle phase involvement. The methodologies 

must be data-driven, with activity effort qualification and quantification. It is 

important to note that the approaches listed in this thesis are usually country or 

service specific. Current methodology for HSI estimates have been performed 

involving risk and benefit assessment, activity/cost factors and categorically 

defined approaches. An example of past cost avoidance methodology research 

1 



includes a study completed by the Canadian organization Defence Technology 
Center (2006). This study provided five arguments with supporting case studies 
recommending Human Factors Integration (Canadian-termed HSI) inclusion into 
cost estimates. These estimates were included early in the acquisition process 
due to development or operational risk impact on programs. The specific 
arguments of the study encompassed reduction of cost (i.e., accident cost, 
redesign costs), technology development (i.e., complexity induced human error), 
identification and mitigation of operational risks (i.e., user involvement to ensure 
project success), resources (i.e., multiple benefit activities) and early involvement 
in project development to ensure success. 

Research is currently ongoing to specify and define HSI activities and cost 
factors for research efforts by civilian and government agencies. A study by Liu, 
Rhodes and Valerdi (2009) investigated including HSI into the cost estimates. Liu 
et al. (2009) provided a case study of the Pratt & Whitney F119 engine, which 
powers the F-22 Raptor fighter aircraft. In this case, the Air Force, in response to 
the need to reduce life cycle cost, created the Reliability, Maintainability and 
Sustainability (RM&S) program in 1984. The RM&S program increased 
requirements for logistics and cost emphasis in the design of the F-22 engines. 
The push allowed for a competition in engine development by General Electric 
(GE) and Pratt & Whitney. GE chose improved F120 engine performance over 
using the RM&S standards. Pratt & Whitney surpassed GE by using integrated 
product development teams (IPT) with group involvement of seven out of nine 
domains of HSI (i.e.. Survivability, Manpower, Personnel, Safety, Training, 
Human Factors Engineering and Occupational Health). The use of the IPTs 
allowed Pratt & Whitney to gain insight from different disciplines, as well as from 
the user population that affected their design decisions. This leads to an 
increased investment of time in the demonstration and internal efforts. Ultimately, 
HSI and other related testing increased 50% during the Engineering and 
Manufacturing Development (E&MD) phase. As a result of Pratt and Whitney’s 
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contribution of a management plan and development schedule, the Air Force 
awarded them the contract. Pratt and Whitney truly demonstrated an 
understanding of RM&S needs. 

The Canadian Defence Technology Center conducted research from 2000 
to 2004 on the application of HSI during 31 Defense acquisition programs 
providing cost to benefit analysis results savings of $3.33M. The methodology 
that the organization used to conduct the research was divided into three 
categories of HSI application based on five domains: Human Factor Engineering, 
Manpower, Personnel, Training, and combined System Safety and Health 
Hazard Assessment. The categories included the immediate saving of time and 
money that occurred during work effort, extrapolated savings (early HSI 
application produced saving money over time), and uncalculated savings 
(decisions resulting in saved lives or increased operational effectiveness). Each 
of these three measures provided definitive terminology and a dictionary of 
activities that when combined, provided savings across the life cycle of any 
program. 

Kopardekar and Hewitt’s (2002) concept paper for the Federal Aviation 
Administration discussed four methods of conducting human factor cost 
estimates involving committees, parametric methods (for human factor impact or 
type of study), and heuristic approaches. The study recommendations were 
based on percentage point system conversions for activities and complexity of 
HSI. At the time of completion, a recommended approach could not be specified, 
with the exception that the formation of committees for consistency of process 
estimation would be beneficial. 

HSI domain cost methodologies are also scattered throughout such 
disciplines as Environmental, Safety and Health (EER System Inc, 1998). The 
cost analysis guides are predecessors of current HSI practices that can be 
updated for today’s terminology and acquisition practices. Together, these 
applications can be modified with existing technology and methodology 


3 



development. Together, these practices could help produce a general HSI 
system cost model and support the initiative to provide HSI costs early in the 
acquisition of systems. 

1. Identification of HSI Domain Concentration in Life Cycie 
Phases 

Each of the previous studies recommended further work to identify, 
quantify or qualify the activities of HSI within the life cycle phases. These HSI 
activities must be specified in order to provide sufficient cost relationships that 
will support cost estimation procedures. Examples of HSI-defined activities in life 
cycle phases are provided by the Defense Acquisition Integrated framework 
(Bahnmaier & Cochrane, 2009), Pacific Science and Engineering Group / Space 
and Naval Warfare Systems (SPAWAR) (Belk, Gepp, Risser, & Smillie, 2009), 
and Lindberg and Carr (2007). The Defense Acquisition Integrated Framework 
provides explicit definitions and procedural guidance throughout the life cycle 
phases. With this guidance, the Naval Postgraduate School has applied HSI 
specific activities for interactive teaching of the Acquisition life cycle that can be 
used for thesis development. These activities occur during each relevant phase 
in the life cycle. The Pacific Science and Engineering Group and SPAWAR have 
developed an integrated framework outlining the HSI activities and their 
interactions with each domain for every phase of the life cycle. Lindberg and Carr 
(2009) present activities by domain and phase that must be considered when 
training future HSI practitioners in Air Force Education and Training commands. 
These activities, the result of weapons system skill analysis, are relevant to cost 
estimates because task performance is based on the level of manpower effort (or 
fully burdened cost). 

Other service-centric examples of activity assessments are also available. 
The Army’s MANPRINT Enterprise-tracking Analyzer tool (META) provides 
activity-domain relationships for various programs by life cycle phase. In the life 
cycle phase, manpower effort level and time duration of HSI practitioners are 
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available. The Air Force HSI Office developed an integrated tool similar to the 
DAU integrated framework (Booz Allen Hamilton Inc., 2009), one that aligns the 
HSI activities with the life cycle phase of DoD. 

All of the processes listed above are in various stages of development and 
are specific to the armed service that funded the creation of the application. 
Currently, cost estimation functions have not been applied to the frameworks. 
These frameworks can contribute to a standard HSI domain activity-driven 
framework that is useful for work breakdown structure (WBS) element or Cost 
Element Structure (CES) development for cost estimation. Furthermore, each of 
these frameworks has yet to be compiled to form a standard process that has 
been adopted by DoD to pursue cost estimates. 

2. Problem Statement 

Given the aforementioned research and prototype applications currently 
under development, a standard cost estimation methodology for HSI -one that 
can easily be manipulated to fit the needs of any system still has not been 
developed. 

The USAF’s 711th Human Performance Wing (HPW) has taken on the 
task from the Cffice of Aerospace Studies of creating cost estimation tools for 
HSI. The 711th HPW began the task by considering HSI cost estimation 
practices during the AoA portion of the Defense acquisition process. For a cost 
estimate to be credible, a necessary condition is that it be comprehensive; i.e., all 
of the HSI domains (manpower, personnel, training, human factors, etc.) must be 
included. A formal process of cost estimation for an AoA should identify, justify 
and estimate all HSI-specific activities. 

This cost estimation process for HSI activities should be coordinated with 
current USAF process stakeholders for Manpower, Personnel, Training, 
Environmental, Safety and Occupational Health. Furthermore, the estimation 
process should integrate all domains of HSI within services to the greatest extent 
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possible, starting with the Capabilities-Based Assessment and continuing 
through Full Operational Capability. Prior to undertaking the development of a 
cost estimation methodology, an HSI Concept Model is needed to establish 
domain relationships within each of the four life-cycle phases (see Figure 1). This 
model correctly identifies those activities used by HSI practitioners. If a viable 
cost estimation methodology is developed, current Air Force programs, such as 
the unmanned aerial system (LIAS) will benefit from results gathered during this 
thesis. 


Life Cycle Cost vs. Time 



Figure 1. Life Cycle Cost (From Nussbaum, 2010) 


3. Analysis of Alternatives Process 

Chairman of the Joint Chief of Staff Instruction (CJCSI) 3170.01 G, (2009) 
documents the general process for establishing life cycle costs during the AoA 
portion of the acquisition process. Capability gaps assessed in the JCIDS 
process allow for three courses of action: 

• Accept the operational risk and perform no action. 
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Seek doctrine change with a non-materiel approach. 
Provide a materiel solution. 






Once the Initial Capabilities Document (ICD) documents the requirements 
for a materiel solution, the Milestone Decision Authority (MDA) determines the 
scope of the subsequent AoA production. Upon approval at the Milestone 
Development Decision (MDD), the AoA study is initiated and will guide the 
Materiel Solution Analysis phase. Further updates to the AoA occur at the 
Technology Development phase and at Milestone B as seen in Figure 2 
(Defense Acquisition University, 2010. p. 95). 


Analytic Support to the 
Defense Acquisition Management System 


( ceA 


ICD 



OBA Oapabilities-Based Assassmant 

lOD Initial Oapabiities Document 

MDD Materiel Development Decision 

AoA Analyeie of ARernatives 

CDD Capability Development Document 


Figure 2. Analytic Support to the Defense Acquisition Management System 

(From Defense Acquisition University, 2010) 


B. OBJECTIVES 

• This thesis will give the USAF’s 711th FIPW a qualitative and 
quantitative view of FISI-specific cost criteria to conduct tradeoff 
considerations when making complex decisions within AoA. 

• The thesis will benefit the AF HPW by developing a cost estimation 
method or process for the AoA. This method is essential for current 
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systems to support budget development and justification over the 
life cycle of system. 

• By providing this method or process, HSI practitioners and other 
acquisition professionals will have improved information that should 
lead to better decisions. This process or method may also be useful 
for other services and agencies within the Department of Defense 
and the Federal government. 

• The method or process will allow HSI practitioners and other 
professionals the capability to assess and mitigate potential risks 
and uncertainties that will influence the future development of 
systems. 

C. RESEARCH QUESTIONS 

• Where do HSI activities fall within the life cycle phases of DoD 
acquisition? 

• What HSI cost estimation methodologies are currently used and 
why? 

• What time durations or level of effort do HSI practitioners consider 
when performing HSI activities? 

• What are the metric definitions used to determine cost factors of 
HSI activities? 

• What levels of complexity must be considered when completing 
HSI cost estimates? 

• What HSI costs are associated with AoAs? 

• What HSI domain tradeoffs must occur to achieve comprehensive 
cost estimates with AoAs? 
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D. HSI CONSIDERATIONS 

Manpower, Personnel, and Training are the domains most often 
considered when estimating cost. However, the domains of Human Factors 
Engineering, Environment, Safety, Occupational Health, Survivability, and 
Habitability will be also be addressed because they are also involved in tradeoffs 
and cost/ benefit analysis. If more cost factors are developed for AoAs, it should 
lead to the acquisition of systems that meet the needs of the Warfighter within 
the cost constraints imposed by the Federal budget. 

The DoD Instruction 5000.02 Enclosure (8) and the DoD HSI Management 
Plan Enclosure (1) describe the variables of HSI that should be included by the 
program manager on various programs. To the extent possible, these variables 
should be integrated and receive tradeoff consideration in terms of the cost of the 
system or materiel solution used. The Air Force HSI Plan states that the activities 
of HSI must be defined and included in total life cycle cost. HSI activities should 
also be traceable and measurable (i.e., level of effort). These domain variables 
are subject to available data, and may not be inclusive: 

Manpower. In advance of contracting for operational support 
services, the PM shall work with the manpower community to 
determine the most efficient and cost-effective mix of DoD 
manpower and contract support. The mix of military, DoD civilian, 
and contract support necessary to operate, maintain, and support 
(to include providing training) the system shall be determined based 
on the Manpower Mix Criteria and reported in the Manpower 
Estimate. Economic analyses used to support workforce mix 
decisions shall use costing tools that account for fully loaded costs - 
i.e., all variable and fixed costs, compensation and non¬ 
compensation costs, current and deferred benefits, cash and in- 
kind benefits. (Department of Defense, 2008c, p. 60; Department of 
Defense, 2009b, p. 4) 

Personnel: To the extent possible, systems shall not require special 
cognitive, physical, or sensory skills beyond that found in the 
specified user population. For those programs that have skill 
requirements that exceed the knowledge, skills, and abilities of 
current military occupational specialties, or that require additional 
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skill indicators or hard-to-fill military occupational specialties, the 
PM shall consult with personnel communities to identify readiness, 
personnel tempo, and funding issues that impact program 
execution. The acquisition strategy and Life-Cycle Sustainment 
Plan should address modifications to the knowledge, skills, and 
abilities of military occupational specialties for system operators, 
maintainors, or support personnel if the modifications have cost or 
schedule issues that could adversely impact program execution. 
The program manager should consider personnel factors such as 
availability, recruitment, skill identifiers, promotion, and assignment. 
The program manager should consider the impact on recruiting, 
retention, promotions, and career progression when establishing 
program costs, and should assess these factors during trade-off 
analyses. (Department of Defense, 2008c, p. 60; Defense 
Acquisition University, 2010) 

Training: The PM shall work with the training community to develop 
options for individual, collective, and joint training for operators, 
maintainors and support personnel, and, where appropriate, base 
training decisions on training effectiveness evaluations. The PM 
shall address the major elements of training, and place special 
emphasis on options that enhance user capabilities, maintain skill 
proficiencies, and reduce individual and collective training costs. 
The PM shall develop training system plans to maximize the use of 
new learning techniques, simulation technology, embedded training 
and distributed learning (DoD Instruction 1322.26 (Reference 
(be))), and instrumentation systems that provide “anytime, 
anyplace” training and reduce the demand on the training 
establishment. Where possible, the PM shall maximize the use of 
simulation-supported embedded training, and the training systems 
shall fully support and mirror the interoperability of the operational 
system. Identify training requirements and cost to maintain/refresh 
knowledge, skills, and abilities, and training or develop personnel 
attributes and training not currently available to the Navy. Conduct 
affordability assessments that include estimates of the TOC of the 
training infrastructure, manpower, and the training associated with 
each approach. (DoD Directive 1322.18 (Reference (bf); 
Department of Defense, 2008c, p. 61; Department of Defense, 
2009b, p. 4) 

Survivability: For systems with missions that might require 
exposure to combat threats, the PM shall address personnel 
survivability issues including protection against fratricide, detection, 
and instantaneous, cumulative, and residual nuclear, biological. 
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and chemical effects; personnel survivability against asymmetric 
threats; the integrity of the crew compartment; and provisions for 
rapid egress when the system is severely damaged or destroyed. 
The PM shall address special equipment or gear needed to sustain 
crew operations in the operational environment, including the 
suitability of equipment intended to enhance personnel survivability 
against asymmetric threats. (Department of Defense, 2008c, p. 61) 

Environmental, Safety and Occupational Health: The PM shall 
ensure that appropriate HSI and ESOH efforts are integrated 
across disciplines and into systems engineering to determine 
system design characteristics that can minimize the risks of acute 
or chronic illness, disability, or death or injury to operators and 
maintainors; and enhance job performance and productivity of the 
personnel who operate, maintain, or support the system. 
(Department of Defense, 2008c, p. 61) Costs can be associated 
with functions within legacy systems increasing accident rates or 
hazardous environments that predispose long-term medical related 
costs. Costs associated with concept development allowing for 
mitigation of safety and health issues (e.g. warning systems, 
enhanced safety procedures) will also be relevant. 

Habitability: The PM shall work with habitability representatives to 
establish requirements for the physical environment (e.g., adequate 
space and temperature control) and, if appropriate, requirements 
for personnel services (e.g., medical and mess) and living 
conditions (e.g., berthing and personal hygiene) for conditions that 
have a direct impact on meeting or sustaining system performance 
or that have such an adverse impact on quality of life and morale 
that recruitment or retention is degraded. (Department of Defense, 
2008c, p. 60) 

Human Factors Engineering: Where practicable and cost effective, 
system designs shall minimize or eliminate system characteristics 
that require excessive cognitive, physical, or sensory skills; entail 
extensive training or workload-intensive tasks; result in mission- 
critical errors; or produce safety or health hazards. Identify MPT 
TOC savings achieved through analysis of design human factors, 
workload, and associated KSAs, resources, and tools for all tasks 
allocated to humans required for the efficient operation, 
employment, and support of the system. (Department of Defense, 
2008c, p. 60; Department of Defense, 2009b, p. 3) 
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E. THESIS ORGANIZATION 

Chapters pertaining to the development of the final product will organize 
this thesis. Chapter II reviews the literature that was used to develop and 
establish the cost estimation process. Chapter II will be broken into six parts. 

Part One reviews the cost estimate decision support for thesis 
development. The section includes the statutory and regulatory basis for cost 
estimates, Department of Defense and life cycle management processes, Joint 
Capabilities Integration Development System (JCIDS) processes and HSI 
relevance within the acquisition life cycle. Part Two reviews cost estimation 
techniques, methods and processes to support thesis development. The section 
discusses the scope, estimate type purposes, rules and assumptions, and 
estimation techniques for this thesis. Further discussion will include modeling and 
identification of cost drivers as well as cost estimation methodologies to be in HSI 
effort determination. Part Three discusses the HSI Concept Model that will be 
used to develop our HSI cost estimation process. Part Four will discuss relevant 
cost estimation methods that are currently used for HSI cost estimation. Part Five 
will cover the system of interest for practical application, validation and 
verification of potential HSI cost estimation processes derived during thesis 
development. 

Chapter III will discuss the method or processes to be used in the analysis 
of data. Chapter IV will cover the results of the analysis of data. Chapter V will 
discuss relevant lessons learned. Finally, Chapter VI will discuss the way 
forward. 
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II. LITERATURE REVIEW 


The process of cost estimation goes beyond merely performing 
calculations for AoA or for congressional reports. Cost estimation is an 
amalgamation of support systems that helps ensure the validity, reproducibility 
and affordability of any defense acquisition program. As stated earlier, the AoA 
starts with Warfighter requirement determination. The requirements must meet 
the policy and strategy guidance, after which they must receive approval from 
Program, Budget and Execution processes before system capabilities can be 
produced. The focus of this chapter is to discuss these processes as well as the 
HSI activities that should be used for cost estimation. This chapter discusses the 
following: 

• Cost estimation decision support system, 

• Cost estimation methods, 

• HSI concept models, 

• Current HSI cost methods. 

A. COST ESTIMATE DECISION SUPPORT SYSTEM 

This section discusses the decision support system and HSI activity 
development in the Life Cycle Management system. Parts one through three 
discuss the decision support system including the Joint Capabilities Integration 
and Development System (JCIDS); Acquisition Management and the DoD 
Planning, and Programming, Budget and Execution (PPBE) process. Part four 
discuss HSI processes in defense acquisition that require funding and/or 
manpower support. These processes are used in the development of a cost 
estimation process for HSI activities. 
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1. statutory and Regulatory Requirements for Cost Estimates 

The requirement for cost estimation is found in the United States Code 
(USC) Title 10, Sections 2430 through and 2435. The USC states that a full life 
cycle cost must be reported for each major defense acquisition program. The 
Secretary of Defense cannot approve the life cycle cost unless an independent 
cost estimate (ICE) of the full life cycle of the program has been completed. The 
ICE must be conducted by entities other than the organization directly 
responsible for the program, regardless of fund source or management control 
(USC, 2006, pp. 1194-1199). USC (2006) section 2433a states that an 
Acquisition Category (ACAT) I program must have approval of the Director of 
Cost Assessment and Program Evaluation (DCAPE) in order to proceed to the 
Technology Development (TD), E&MD, and Production and Deployment (P&D) 
stages. 

Key regulatory costing events or items are mandatory inputs to the 
milestone decisions that are made at the end of the TD, E&MD, and P&D stages. 
These events are as follows: Cost Analysis Requirements Description (CARD), 
Life Cycle Cost Estimates (LCCE) and. Service Cost Position (SCP) / 
Independent Cost Estimate (ICE) (See Figure 3). 

The CARD provides a common technical description and a system 
baseline, as well as technical descriptions for all organizations providing cost 
projections (Nussbaum, 2010). The CARD is reviewed at each milestone. The 
next section discusses the LCCE and ICE cost events. 
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Figure 3. Key Regulatory Costing Events leading to a DAB Milestone 

Decision (From Nussbaum, 2010) 


2. Department of Defense/Life Cycle Management Process 

DoDI 5000.02 and DoD Directive 5000.04 provide further clarification for 
development of LCCE, ICE requirements for ACAT programs. Details for 
establishment of the Office of Cost Assessment (OCA) and Director of Cost 
Assessment and Program Evaluation (DCAPE) are also provided. 

When a program is initiated, the service program office must produce a 
Program Office Estimate (POE). If ACAT ID programs involve significant risk or 
are considered highly visible, a Component Cost Analysis (CCA) will also be 
prepared. The Component Acquisition Executive requests the CCA. Specifically, 
DoDI 5000.02 states: 

For ACAT ID or ACAT IC programs, the independent cost estimate 
is required prior to certification at Milestone A (see 10 U.S.C. 
2366a), Milestone B (see 10 U.S.C. 2366b), and before any 
decision to enter into low-rate initial production or full-rate 
production. Independent cost estimates are also required in 
advance of certification following critical cost growth in major 
defense programs (pursuant to 10 U.S.C. 2433a). In addition, the 
Office of Cost Assessment for an ACAT ID program may conduct 
an independent cost estimate at any other time as directed by 
either the DCAPE or the USD (AT&L). (Defense Acquisition 
University, 2010; Department of Defense, 2008c) 
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Although an ICE is not required for ACAT lA programs, the DoD 5000 
series requires a CCA at Milestone B and whenever an Economic Analysis is 
requested by the Milestone Decision Authority (MDA). The CCA and Economic 
Analysis estimates are completed by the OSD Office of Cost Assessment staff or 
sponsoring DoD components. Furthermore, milestone and decision reviews 
require a POE. However, life cycle benefits and costs, which are subject to the 
program’s integrated product team approval, are required for the ACAT lA POE 
(Department of Defense, 2008c). 

DoDI 5000.02 indicates that all DoD components (Military Departments 
and Defense Agencies) prepare LCCEs in support of their acquisition programs. 
“A LCCE attempts to identify all the costs of an acquisition program, from its 
initiation through disposal of the resulting system at the end of its useful life and 
to properly phase, or spread, the costs for inclusion in budget submission 
documents” (Department of Defense, 2008c; Defense Acquisition University, 
2010). To account for affordability, the LCCEs are used during the review 
process to ensure system affordability and to provide budgetary considerations 
for Congress. Integrated Product Teams (IPTs) and an Integrated Product and 
Process Development (IPPD) are typically employed to ensure LCCEs are 
completed accurately. LCCEs are used in the Planning, Program, Budgeting and 
Execution process (PBBE) which, in turn, is used for the funding requests that 
are sent to Congress. 

The last regulatory action to discuss is the establishment of the Office of 
Cost Assessment (OCA) and the DCAPE. DoD Directive 5000.04 states that “the 
OSD OCA provides independent analysis and advice to DoD officials on matters 
of cost estimation and cost analysis for weapons acquisition programs, including 
matters of program life cycle cost. The DCAPE provides policies and procedures 
for the conduct of all DoD cost estimates. The DCAPE also issues guidance 
relating to the full consideration of life-cycle management and sustainability 
costs. In addition, the DCAPE reviews DoD Component cost estimates and cost 
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analyses conducted in connection with Major Defense Acquisition Programs 
(MDAP) and Major Automated Information Systems (MAIS)” (Department of 
Defense, 2008b). 

3. Joint Capabilities Integration Development System (JCIDS) 

This section discusses the principles that underlie JCIDS. Implemented in 
2002, the JCIDS process evolved because the requirements process was flawed. 
The three underlying principles of the JCIDS process include: (a) Determining the 
needs from a joint perspective, (b) describing those needs in terms of 
capabilities, and (c) overseeing the DoD functional portfolio by a flag officer. 
These principles lead to the identification of the capabilities needed rather than to 
a specific materiel solution (i.e., system function need versus system 
requirement). A joint perspective allows for system development instead of 
parallel development, which may produce duplicate costs. Finally, the process, 
overseen by a flag officer, allows the scope of the solution to meet joint service 
needs rather than narrow requirements suiting one function. 

The JCIDS process was revised in 2009 to include a Capability Based 
Assessment (CBA). The revision removed the processes of Functional Area 
Analysis, Functional Needs Analysis and the Functional Solutions Analysis. As 
the entry point into the acquisition process, the CBA identifies gaps and provides 
recommendations for closing the gap. As stated in Chapter I, this process 
eventually leads to a DOTMLPF Change Recommendation (DCR) and/or an AoA 
for a materiel and/or non-materiel solution. 

It is important to consider HSI early in order to mitigate total cost. HSI 
consideration begins during the CBA when HSI subject-matter experts provide 
input for activities that will eventually be inserted into the Initial Capabilities 
Document (ICD) and Capabilities Development Document (CDD) (See Figure 4, 
Simpkiss, 2009). A CBA should consider the full spectrum of solutions, including 
doctrine, organization, training, materiel, leadership and education, personnel. 
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and facilities (DOTMLPF). If the ICD recommends a materiel solution, an AoA 
study plan is created. HSI practitioners can contribute to any of these 
alternatives. 



Figure 4. Human System Integration in Requirements (From Simpkiss, 2009) 

Once the CBA has been completed, an ICD is written. Warfighter needs 
are identified in the JCIDS process and are developed into operational terms for 
the ICD. These operational terms are then molded into specific requirements in 
the CDD. The Joint Requirements Oversight Committee (JROC) and the Defense 
Acquisition Board (DAB) monitor and review the JCIDS activities throughout the 
acquisition life cycle. 

The past three sections covered statutory and regulatory requirements for 
cost estimation, the Life Cycle Management Process and the Joint Capabilities 
Integration and Development System. The next section discusses the HSI 
activities that should be considered part of the cost estimation activities that 
occur in the acquisition process. 
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4. 


Human System Integration in the Acquisition Life Cycie 


The Air Force HSI Requirements Pocket Guide (2009) and current service 
policies provide an excellent overview of the HSI activities in each phase of the 
acquisition life cycle. These documents will be used to explain the relevant HSI 
activities in each phase and their associated cost metrics. The Air Force Pocket 
Guide (2009) provides a detailed description of the HSI effort within the 
Integrated Defense Acquisition, Technology, and Logistics Life Cycle 
Management System framework. These policies were chosen because of the 
abundance of use in the HSI community. Other policies exist such as MANPRINT 
(AR 602-2). The MANPRINT Handbook was review but did not add any 
additional insights to the present discussion. 

Each military service has its own policies that mandate how HSI should be 
accomplished in the acquisition process. Four current service policies that are 
particularly informative are the (a) OPNAVINST 5310.23, Navy Personnel Human 
System Integration (NAVPRINT), (b) Air Force Instruction (AFI) 10-601, 
Capabilities Based Requirements Development, (c) AFI 63-1201, Life Cycle 
Systems Engineering and (d) AFI 63-101, Operations of Capability Based 
Acquisition Systems. 

OPNAVINST 5310.23D Enclosure (1) specifies that: 

The CBA must account for current and projected manpower and 
personnel characteristics. CBA analysis must identify operational or 
environmental conditions that may impact performance of the 
operators, maintainors and support personnel for future platforms 
and systems. 

The AoA provides the foundation for KPPs, key system attributes 
(KSA), and other attributes in CDDs and CPDs; therefore it is vitally 
important that HSI considerations, where applicable, be evaluated 
during the AoA. This allows HSI considerations (e.g., manpower 
and systems training) and their associated costs, to be addressed 
when selecting the preferred alternative. (Department of Defense, 

2009e) 
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The Department of the Air Force Instruction, AFI 63-1201 emphasizes the 
incorporation of MSI into the System Engineering Plan (SEP) with specific 
guidance for Environment, Safety, Occupational Health and Habitability 
requirements. Specifically addressed are the Maintenance and Support 
personnel, who are the backbone of any operation. The document states: “It 
must be addressed throughout the life cycle, and must be consistently integrated 
into SE implementation to balance total system performance (hardware, 
software, and human), assurance of Operational Safety, Suitability, and 
Effectiveness (OSS&E), survivability, and affordability” (Department of Defense, 
2007a). 

Air Force Instruction 63-101 (2009) states that “the HSI office will oversee 
and advocate an HSI focus in activities regarding systems integration, systems 
engineering, total system performance and total operating costs” (pg. 41). This 
instruction highlights the system engineering process by stating: “ESOH 
integration strategy must define HSI roles and activities for those domains” 
(Department of Defense, 2009a). Emphasizing trade-offs between HSI domains, 
the ESOH integration strategy incorporates survivability of the system. In 
accordance with these instructions, the survivability of the system should be 
considered during trade-off analysis when considering affordability, schedule, 
and performance capabilities. 

Clearly, an HSI focus is mandatory for Navy and Air Force acquisition 
programs. The next step is to examine the HSI activities that occur during each 
phase of the acquisition process and to consider the cost associated with these 
activities. 


a. Materiel Solution Analysis (MSA) Phase 

The focus of the MSA phase is ‘completing Analysis of Alternatives 
to assess potential materiel solutions to capability need, identifying 
key technologies and estimate life cycle costs’. This phase 
‘considers commercial-off-the-shelf solutions from both large and 
small business and identifies materiel solution to capability need’. 
(Defense Acquisition University Online, 2010) 
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The overall goal of the MSA phase is to identify an affordable, 
operationally effective materiel solution. During this phase, inputs are provided 
for MDA cost estimates, AoA study plan/analysis, study contracts. Technology 
Development Strategy (TDS), Request for Proposals (RFP) and draft CDD 
(Department of Defense, 2008c). The cost estimates given during this phase 
allow the MDA to determine adequate funding of the Technology Development 
Phase for the preferred materiel solution. In this section, relevant HSI activities 
are described from current resources including: (a) SECNAV-5000.2 Navy 
Acquisition and Capabilities Guidebook, (b) Defense Acquisition Guidebook, (c) 
DoD 5000 series instruction, (d) 2010 Weapon System Reform Act (WSARA) 
and the (e) Human Systems Engineering Manual. 


SECNAVM-5000.2 states the importance of HSI activities within the 
Technology Development Strategy (TDS) (See Figure 5). The TDS includes: (a) 
Determining issues related to physical/mental characteristics, (b) describing the 
target audience description (TAD), and (c) providing performance parameters for 
operators and maintainers. When determining affordability, feasibility, and risk, 
HSI practitioners must also consider training, manpower, job specialization 
(Personnel Rates/MOS), and technology insertion (DoD [2008d]; Enclosure 7). 
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Interoperability Summary 
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• Industrial Capability & Manufacturing 
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• Business Strategy 



Figure 5. Technology Development Strategy Contents (From Defense 

Acquisition University, 2010) 
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The second resource is the Defense Acquisition Guidebook (DAG), 
Chapter 3. The DAG describes the AoA process and begins with a detailed 
structure of the AoA Study Plan. Included in the plan are analysis for scenario 
and threat development (bound by physical and environmental constraints). 
Capability ranges are also used to determine testing scenarios, while operation 
effectiveness measures are used to associate metrics for military worth with the 
alternatives (i.e., mission outcome, system performance, identification of 
simulation models). 


Life-cycle cost analysis will be a part of the study plan. The life 
cycle cost estimate is combined with the operational effectiveness analysis for a 
full cost-effectiveness comparison portrayal (see Figure 6). The final segment of 
the AoA Study Plan includes the organization and management structure for 
conducting the AoA, as well as any oversight and guidance needed. With the 
AoA Study Plan in place, the AoA can proceed as described in the next 


paragraph. 


Notional Cost-Effectiveness Analysis: 
Display of Results 


R«ctangl«B indicat* uppar and iowar bounds of 
sensitivfty anat/ses (or Dounds for distrioution of 
results from stochastic models) 


Alt 6 is more cost-effective 
than 5 or 4: at lecet as 
cfTective but at lower cost 



Figure 6. Sample Scatter Plot of Effectiveness versus Cost comparison 
(From Defense Acquisition University, 2010) 
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The HSI activities for the AoA process are distributed throughout 
DoD 5000 series instructions. Possible AoA materiel solutions are based on the 
ability for system alternatives to meet/fulfill concepts of operation, measures of 
effectiveness, cost, schedule, and risk criteria for technologies. Commercial-off- 
the-Shelf (COTS) solutions are used if possible. DoDI 5000.02 directs that “the 
AoA shall assess the critical technology elements associated with each proposed 
materiel solution, including technology maturity, integration risk, manufacturing 
feasibility, and, where necessary, technology maturation, and demonstration 
needs” (Department of Defense, 2008c). 

The 2010 Weapons System Acquisition Reform Act (WSARA) 
explicitly amends DoDI 5000.02 and the Defense Federal Acquisition Regulation 
Supplement to require competition for prototype development. Fox (2010) stated 
that the 2010 WSARA will help; 

• Identify cost risk earlier, 

• Baseline programs with realistic cost and schedule estimates, 

• Ensure rigors of AoA, 

• Ensure Departments fully fund development and procurement cost 
of programs, and 

• Lead Departments in improving cost assessment and analysis. 

The implementation of the 2010 WSARA affects performance assessments of an 
AoA by requiring that all acquisition programs: 

Evaluate the cost, schedule, and performance of the program, 
relative to current metrics, performance requirements, and baseline 
parameters, and determine the extent to which the level of program 
cost, schedule, and performance, relative to established metrics, is 
likely to result in the timely delivery of a level of capability to the 
Warfighter that is consistent with the level of resources to be 
expended and to provide superior value to alternative approaches 
that may be available to meet the same requirement. (Hansen, 

2010 ) 
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In performing AoA studies, the Navy and Air Force identify trade-off 
analysis as a key factor for HSI involvement. The Navy Acquisition gate process 
(Whitaker, 2009) (See Figure 7) identifies capability gaps and applies acceptable 
operational thresholds and objectives when conducting the trade-off studies. At 
the Gate 2 review, analysis of DOTMLPF must occur. The HSI practitioner can 
contribute by assessing tradeoffs among HSI domains and providing specific cost 
adjustments to mitigate cost overruns. Practitioners also identify capability gaps 
to provide input for rough order of magnitude cost assessment as specified in 
OPNAV Instruction 5310.23. Capability gap contributions and/or limitations may 
include: 

• Manpower, personnel, or training (including military, contractor, 
civilian and recruitment/retention of each), 

• Contributions of operator, maintainer, support personnel, total 
system performance (including infrastructure), 

• Requirements for safety, survivability, habitability, and 

• System reliability, maintainability, transportability, and supportability 

A goal of capability gap identification is the reduction of the system’s logistics 
footprint and total ownership cost (Shattuck, 2010). 
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Acquisition Governance Board Tiger Team Draft Gate Alignment 2.0 
as of 08/03/09 DODI 5000.02 (Dec 2008) 

Program Initiation at Milestone B 
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Figure 7. Navy Gate Alignment Dec 2008 (From Whitaker, 2009) 


The aforementioned gap analyses are an important exit criteria for 
the materiel solution analysis phase and are assessed at Milestone A prior to the 
development of the system. Furthermore, the use of the capabilities assessments 
will allow comparisons with benchmark resources (current system capabilities 
and costs) and future developments in AoA. 

The Human System Engineering (HSE) Manual provides further 
guidance for HSI activities and cost considerations during the MSA phase. 
Insights into potential COTS materiel solution approaches and HSI domain trade- 
space analysis are specified. The HSE Manual states: 

The COTS approach has (a) the lowest projected life cycle cost, 
within acceptable risks, and meets essential requirements, 
including human performance requirements; (b) the lowest human 
workload and manning requirements for operations and 
maintenance; (c) the most effective training program; and (d) the 
least safety and health hazards. (Naval Sea Systems Command, 

2009, p. 29) 
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Domain trade-offs include optimizing manpower, addressing 
training requirements, incorporating safety, and designing for survivability. In 
conducting trade-offs, manpower can be optimized by including the proper mix of 
personnel, conducting a workload analysis, reviewing policy, and increasing 
performer effectiveness. Training requirements for specific designs are also 
considered. Risk reduction and hazard prevention in system design must 
consider safety and health concerns. In trade-off studies, the personnel 
survivability equipment design is an essential consideration. Increased 
performance and maintenance requirements are used to research and test 
equipment design. The Life Cycle Sustainment Plan (LCSP) also recognizes 
these domain initiatives. The LCSP considers the domain trade-offs as 
contributing factors to the reduction of total operating costs. 

Finally, habitability requirements for personnel are considered, but 
generally are the least found in trade-off studies. Habitability trade-offs may 
include design of living conditions that would impact working conditions and 
quality of life. Thus far, section (a) has considered HSI activity development and 
trade-off studies in the MSA phase. The next step concerns the technical review 
process that ensures performance and funding requirements are met. 

The MSA phase includes two technical review processes to ensure 
programs meet performance and funding requirements. These technical reviews 
are the Initial Technical Review (ITR) and the Alternative System Review (ASR). 
The ITR ensures cost estimate validity, assesses capability needs and materiel 
solutions, and identifies cost drivers. As stated in the NAVSEA HSE manual, the 
ITR process may include HSI assessments based on mission scenario functions 
allocated by the Navy Tactical Task list. By doing so, performance metrics are 
identified and thus, drivers (i.e., manpower effort) for cost relationships are 
established. ITR scenarios may also identify training requirements imposed by 
system operation. 
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The ASR, on the other hand, ensures that the system is ready to 
move on to the Technology Development phase and meets the requirements set 
by the user. The ASR will provide a review of the AoA and provide a 
comprehensive rationale for the proposed solution (Shattuck, 2010). By 
reviewing alternative materiel solutions, the ASR helps: 

... ensure that sufficient effort has been given to conducting trade 
studies that consider and incorporate alternative system designs 
that may more effectively and efficiently meet the defined 
capabilities. (DAU Online, 2010) 

The HSI practitioner aids in this process by being involved in the Integrated 
Product Team and by providing input for workload and manpower requirements 
for each alternative. By doing so, performance effectiveness, training 
assessments, system interoperability, test and evaluation (T&E) and life cycle 
costs (including schedule, cost, and performance trade-offs) for each alternative 
reflects actual costs more accurately. 

The Materiel Solution Analysis phase ends when the exit criteria 
are met, including: 

• Initial acquisition strategy (including the sustainment strategy), 

• Contractual documents required to continue into the Technology 
Development Phase, 

• Initial support and maintenance concepts, 

• Life cycle costs, and 

• Manpower estimates for the system concept. 

Finally, input is provided for the draft CDD and the Test & Evaluation Strategy 
(TES). The TES must include estimated costs pertaining to system operation and 
system attributes. HSI analysis will be included in the TES and must be included 
in cost estimates. 
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Section (a) provided an overview of HSI activities in the Materiel 
Solution Analysis phase. See Table 1 for a summary of HSI activities during the 
MSA phase. HSI manpower effort and/or HSI funding requirements were 
established. Section (b) will provide an overview of the same issues for the 
Technology Development phase. 


Table 1. MSA Phase HSI Activity Summary 


HSI Activity 

Publication 

IDAT&L Activity 

1. Determine issues related to 
physical/mental characteristics. 

SECNAVM-5000.2 

Technology Development 
Strategy (TDS) 

2. Describe the target audience description 
(TAD). 

SECNAVM-5000.2 

TDS 

3. Provide performance parameters for 
operators and maintainers. 

SECNAVM-5000.2 

TDS 

4. Assess affordability, feasibility and risk. 
Consider training, manpower, job 
specialization (Personnel Rates/MOS), and 
technology insertion (Enclosure 7). 

SECNAVM-5000.2 

TDS 

5. Conduct scenario and threat development 
analysis as well as capability ranges. 

Defense 

Acquisition 

Guidebook 

AoA Study Plan 

6. Assess the critical technology elements. 

DoD 5000 Series 

AoA 

7. Review DOTMLPF. 

8. Conduct trade-off analysis of among HSI 
domains and provide specific cost 
adjustments to mitigate cost overruns. 

2010 WSARA 

AoA 

9. Identify capability gaps to provide input for 
rough order of magnitude cost assessment. 

OPNAV Instruction 
5310.23 

AoA 

10. Conduct a COTs HSI domain trade-space 
analysis. 

HSE Manual 

AoA 

11. Review HSI mission scenario 
assessments, metrics, training requirements 

HSE Manual 

Initial Technical Review 
(ITR) 

12. Participate in Integrated Product Team; 
Provide input for workload and manpower 
requirements for each alternative. 

Shattuck, 2009 

Alternative System Review 
(ASR) 

13. Provide input for draft CDD and the Test 
and Evaluation Strategy (TES) pertaining to 
system operation and system attributes. 

Shattuck, 2009 

Alternative System Review 
(ASR) 
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b. Technology Development Phase 

The focus of this phase is ‘to reduce technology risk and to 
determine the appropriate set of technologies to be integrated into 
the full system. This effort is normally funded only for advanced 
development work and does not mean that a new acquisition 
program has been initiated’. (DAU Online, 2010) 

DoDI 5000.02 and the JCIDS Manual (2009) state the overall goal 
of the Technology Development (TD) phase is to reduce risk and determine the 
maturity of systems for full system integration and prototype development. The 
TD phase contributes to continuous technology discovery and successful 
collaboration between the user, the S&T community, and the system developer. 
Viability of technologies and refining user requirements is of utmost importance. 
During this phase, inputs are required for the following documents and activities: 

• Capability Development Document (CDD), 

• Acquisition Strategy (AS), 

• Acquisition Program Baseline (APB), 

• System Engineering Plan (SEP), 

• Test and Evaluation Master Plan (TEMP), 

• Request for Proposal (RFP), 

• System Performance Specifications, and 

• Technical Reviews 

In this section, the following resources will be used to describe the 
TD phase HSI activities: 

• DoD 5000 series instruction, 

• JCIDS Manual, 

• OPNAVINST 5310.23, 
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• 711**^ Human Performance Wing Human System Integration 

Development (HPW HSI) Guide, 

• Human Systems Engineering (HSE) Manual, and 

• Secretary of the Navy (SECNAV) guidance 

HSI practitioner involvement in the TD phase begins with the 
development of the Capability Development Document (CDD). DoDI 5000.02 
states that every acquisition program or evolutionary increment requires a CDD. 
The CDD shall refine integrated architecture as well as clarify how the program 
will lead to joint warfighting capability. Additionally, the JCIDS Manual states that 
the CDD must address the system capabilities and key performance parameters 
(KPPs) of the system. HSI issues that affect the suitability, affordability and 
effectiveness of the system are also addressed (Department of Defense, 2008c). 
The JCIDS manual specifies that design considerations must consider 
environmental factors, unplanned stimuli and mission capability. Mission 
capability is extremely important especially in nuclear, biological, chemical, and 
environments. 

OPNAVINST 5310.23 describes the HSI-related capabilities that 
should be discussed in the CDD. The KPPs should include threshold and 
objective values that ensure Warfighters are able to achieve required levels of 
performance. The 711*^^ HPW HSI Guide and HSE Manual address the human- 
centered KPPs and capabilities that should be considered in system design. 

HSI practitioners also provide input into the Acquisition Strategy 
(AS) and Acquisition Program Baselines (APBs). HSI input (as part of an IPT) is 
governed by SECNAV guidance and service policies. SECNAVINST 5000.2D 
indicates: 

Life cycle cost projections for capabilities and/or systems shall 
include direct HSI costs (e.g., MPT), and should discuss indirect 
costs (e.g., medical benefits resulting from safety and occupational 
health risks)...HSI issues and domains must be considered to 
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ensure that configuration changes do not create new or unforeseen 
HSI issues. (Department of Defense, 2008d, p. 12) 

Milestone B approval is dependent on the Acquisition Strategy which will 
incorporate development, testing, production and life cycle support as critical 
elements. 

APBs are a necessary step in establishing the trade-offs between 
cost, schedule and performance. Many of the KPP, KSA, threshold/objective 
values, and requirements are human performance based, which affect system 
performance. Within the APB, operationally determined performance parameters, 
measures of effectiveness, and performance help to determine drivers cost 
estimates. The APB consists of the following components (see Table 2): 


Table 2. APB Cost Involvement (From Shattuck, 2009) 


Research, development, test and 
evaluation costs 

Operations and maintenance (O&M) 
costs 

Procurement costs 

Average Procurement Unit 

Military construction costs 

Program Acquisition Unit Cost 

Total system quality 

Any other cost objectives established by 
the milestone Decision Authority 


The APB is approved by the MDA and must concur with the Program Executive 
Officer (PEO) for all ACAT programs, and the DoD Component Acquisition 
Executive (CAE) for ACAT ID and lAM programs (Shattuck, 2009). 

The HSE Manual describes the application of HSI initiatives in the 
Acquisition Strategy, System Engineering Plan, Test and Evaluation Master Plan 
and Request for Proposal. To fulfill the HSI objectives of the system, the manual 
incorporates the following: 

• Human performance, 

• Workload, 

• Safety issues. 
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• Risk assessments, and 

• Reduction in technology applications 

HSI practitioners also provide input on human techniques, system interactions 
(i.e., software, hardware) and HSI metrics. The activities will emphasize “mission 
capability, readiness, force structure, affordability, performance effectiveness, 
and achievement of wartime operational objectives” (NAVSEA, 2009, pp. 6-51). 
By doing so, HSI needs and requirements are successfully included in 
technology contracts and prototype RFPs. 

A major output of the TD phase is the system performance 
specifications. HSI practitioners provide input to the program manager to help 
determine trade-offs for cost, schedule and performance. The HSE Manual 
addresses performance specifications by identifying human roles and 
responsibilities. The manual includes “Requirements for sustained human 
performance...prevention of human error...approaches that reduce human error 
and cognitive workload and human machine interface designs that facilitate 
human performance” (Shattuck, 2009, p.4; NAVSEA, 2009, pp.6-55). 

Finally, three major reviews occur in the TD phase. These reviews 
are the System Requirement Review (SRR), System Functional Review (SFR) 
and the Preliminary Design Review (PDR). HSI input into each of these reviews 
allows for the mitigation of potential shortfalls and the survivability of the system 
in future operational testing, evaluation and verification events. 

The SRR specifies the allocation of tasks to the human, software, 
and hardware in the system. The SFR includes operational level functional 
requirements for the various subsystems. The review also allows for assignment 
of those requirements to operations, maintenance and sustainment 
considerations. IPTs can then initiate preliminary design. 

The PDR establishes a “physically allocated baseline” that helps 
determine the level of operational effectiveness that the system will demonstrate 
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(Shattuck, 2009). The PDR ensures design configuration and capabilities are in 
accordance with cost, schedule, and performance requirements. The PDR may 
be completed in the Engineering and Manufacturing Development (E&MD) 
phase. The TD phase ends at Milestone B and at the completion of the 
preliminary design. 

Section (b) provided an overview of HSI activities in the Technology 
Development phase. Table 3 provides a summary of the HSI activities during the 
TD phase. Section (c) will provide an overview of the same issues for the 
Engineering and Manufacturing Development phase. 
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Table 3. TD Phase HSI Activity Summary 


HSI Activity 

Publication 

IDAT&L Activity 

1. Address the system capabilities 
and key performance parameters 
(KPPs). 

2. Assess the suitability, affordability 
and effectiveness of the system 
(consider environmental factors, 
unplanned stimuli and mission 
capability). 

DoDI 5000.02; 
JCIDS Manual; 
OPNAVINST 
5310.23; 711‘^ 
HPW HSI Guide 
and HSE 

Manual 

Capability 

Development 

Document (CDD), 

3. Life cycle cost projections for 
capabilities and/or systems shall 
include direct HSI cost. 

4. HSI issues and domains must be 
considered to ensure that 
configuration changes do not create 
new or unforeseen HSI issues. 

SECNAVINST 
5000.2D; 
service policies 

Acquisition Strategy 
(AS), 

5. Assess KPP, KSA, 
threshold/objective values, and 
requirements for trade-off analysis. 

SECNAVINST 
5000.2D; 
service policies 

Acquisition Program 
Baseline (APB), 

6. Assess system for human 
performance, workload, safety issues, 
risk assessments, and reduction in 
technology applications. 

7. Provide input on human 
techniques, system interactions (i.e., 
software, hardware) and HSI metrics. 

HSE Manual 

System Engineering 
Plan (SEP), Test and 
Evaluation Master 

Plan (TEMP), and 

8. Assess HSI needs and 
requirements in the technology 
contracts. 

HSE Manual 

Request for Proposal 
(RFP). 

9. Provide input to the PM to 
determine trade-offs for cost, 
schedule and performance. 

HSE Manual 

System performance 
specifications 

10. Provide input into the technical 
reviews for mitigation of potential 
shortfalls and the survivability of the 
system in future operational testing, 
evaluation and verification events. 

HSE Manual 

System Requirement 
Review (SRR), 

System Functional 
Review (SFR) and the 
Preliminary Design 
Review (PDR). 
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c. Engineering and Manufacturing Deveiopment Phase 

The purpose of the E&MD Phase is to develop a system or an 
increment of capability; complete full system integration; develop an 
affordable and executable manufacturing process; ensure 
operational supportability with particular attention to minimizing the 
logistics footprint; implement human systems integration (HSI); 
design for producibility and ensure affordability; protect CPI by 
implementing appropriate techniques such as anti-tamper; and 
demonstrate system integration, interoperability, safety, and utility. 

(DAU Online, 2010) 

The Engineering and Manufacturing (E&MD) phase consists of two 
efforts: the Integrated System Design (ISD) and the System Capability and 
Manufacturing Process Demonstration (SC&MPD). The E&MD phase begins 
after a successful Milestone B review and includes a Post Critical Design Review 
(CDR) Assessment at the end of the ISD. The Post-CDR Assessment will 
confirm design maturity and provide the initial product baseline. The following 
P&D phase documents guide the E&MD phase: 

• Capability Development Document, 

• Acquisition Strategy, 

• System Engineering Plan, and 

• Test Evaluation Master Plan (TEMP) 

The SC&MPD effort, completed after the Post-Critical Design Review, shall 
demonstrate the operation of the systems (with the approved KPPs) in the 
intended environment of use. 

The E&MD phase contains six embedded processes (i.e., JCIDS, 
Oversight and review. Contracting, Major Products, Logistics/ Sustainment and 
Technical). HSI practitioners are not directly involved in production and 
manufacturing activities; however, practitioners do monitor the activities to 
reduce risk. The initial phase activity includes MDA updates to the Acquisition 
Strategy and APB administrative review processes that should contain HSI input. 
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This Input ensures that a defense acquisition program continues to have an HSI 
emphasis and that program managers will be able to maintain planned 
schedules. HSI considerations are important as the E&MD continues with the 
Source Selection Evaluation Board (SSEB). HSI board members will review 
RFPs for HSI concerns. A majority of HSI involvement occurs during prototype 
development. 

In prototype development, HSI practitioners help to build and use 
the models and simulation tools for testing purposes. By doing so, human 
performance modeling allows mitigation of risk and safety issues that can occur 
early in the system development. Further HSI activities revolve around the 
Product Support Plan (PSP) and the Performance-Based Logistics (PBL) 
Strategy. These activities focus on several HSI domains: Manpower, Training, 
Personnel, HFE and Habitability. Because the system may change operationally, 
by mission capability, or by change in organization and/or funding, the logistician 
must update the PSP repeatedly. Therefore, HSI practitioners are continuously 
required to provide input. The DoDI 5000.02, DAG (2010) and Shattuck (2009) 
state that these inputs to include: 

• Reduction of time in maintenance and / or operation due to HFE 
influenced design, 

• Requirement development during Personnel Selection (i.e., 
knowledge, skills, attribute, and experience levels), 

• Manpower reduction or increase use of the proper mix of military, 
contract, civilian workers, 

• Training development, test evaluation and verification, 

• Habitability concerns such as “morale, safety, health, comfort, 
personnel performance, or unit readiness, that could lead to 
recruitment or retention problems” (Shattuck, 2009), and 
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• Environment, Safety and Operational Health concerns such as 
mitigation of health deterioration or mishaps due to system design 
and human centered environmental designs. 

While habitability is important in design considerations, the domain 
criteria usually fall under other guidance. References to publications that guide 
habitability design efforts include; (a) the Naval Sea Systems Command (2006) 
Joint Fleet Maintenance Manual (COMFLTFORCOMINST 4790.3), and (b) the 
Air Force Air Force Habitability and Occupational Health publications. These 
publications are service and/or system specific and refer to the following 
guidance: 

• OPNAVINST 9640.1 , Shipboard Habitability Program, 

• NAVSEA SL720-AA-MAN-010/V1R2. Fleet Modernization Program 
(FMP) Management And Operations Manual 

• NAVSEA SL720-AA-MAN-030 . Surface Ship and Carriers Entitled 
Process for Modernization Management and Operations Manual 

• Air Force Policy Directive 23-1 (2009), Materiel Management 
Policies and Procedures, 

• Air Force Policy Directive 21-1 (2003), Air and Space Maintenance, 

• Air Force Occupational Safety And Health Standard 48-20 (2006), 
Occupational Noise And Hearing Conservation Program, 

• Air Force Handbook 21-130, Technical Analysis To Determine 
Criterion For 2 Vs 3 Level Repair, 

• Air Force Occupational Safety And Health Standard 48-139 (1999), 
Laser Radiation Protection Program, 

• Air Force Pamphlet 90-902 (2000), Operational Risk Management 
(ORM) Guidelines And Tools, 

• Preliminary Hazard Analysis, Hazard Listing, Risk Assessment, and 
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• Air Force Instruction 90-1301(2008), Implementing Military Flight 
Operations Quality Assurance. 

The CPD is one of the final documents produced during the E&MD 
phase. The CPD helps to determine the operational performance parameters. 
The CPD is revised as lessons are learned during system development and as 
KPPs and KSAs are refined. The ICD, CDD, AoA, CDR, development and 
operational testing, DoD Enterprise Architecture, and DOTMLPF 
recommendations also influence the CPD. Once approved by the JROC for 
operational acceptance, the MDA can initiate Milestone C and approve entry into 
the Production and Deployment (P&D) phase. 

Military service guidance for incorporating HSI requirements into 
the CPD include the (a) Navy OPNAVINST 5310.23, (b) Air Force 711‘^ HSI 
Development Guide, and the (c) Army MANPRINT documents. Examples of HSI 
requirement activities include: 

• Identify the integrated training system requirements for individual, 
collective, joint, and fleet training support (e.g.. Total Ship Training 
System for ships). 

• Describe in measurable and testable terms, when relevant, the 
missions, functions, or attributes used to optimize manpower, 
personnel readiness requirements. 

• Identify MPT Total Ownership Cost (TOC) savings achieved 
through analysis of design (human factors, workload, KSAs, 
resources, and tools) for all tasks allocated to humans required for 
the efficient operation, employment, and support of the system. 

• Describe any safety, occupational health, and environmental 
compliance requirements that reduce the risk of fatalities, injury, 
illness, or disability and death of the operators, maintainors, and 
support personnel. 
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• Describe habitability requirements for facilities (berthing, personal 
stowage, food service, medical, religious, security, recreational and 
lounge spaces), and ambient environment requirements (e.g., 
noise, lighting, heating, air conditioning, and ventilation, workspace 
layout, etc.) (Department of Defense, 2009e) 

Finally, several System Engineering (SE) Reviews occur prior to 
exiting the E&MD phase. The SE Reviews include the Test Readiness Review 
(TRR), the Production Readiness Review (PRR), the System Verification Review 
(SVR), and the Functional Configuration Audit (FCA). In all, the reviews are in 
place to ensure the designed system meets the specified requirements and 
capabilities but also, to allow Low Rate Initial Production and Full Rate 
Production to occur. As members of an Integrated Product Team, HSI 
practitioners are fundamental to the success of these reviews. HSI input will 
ensure the designs are functional from the users’ perspective with minimal risk to 
them. 

The Technical Outputs are similar to those in the Technology 
Development phase; however, an Initial Product Baseline (IPB) is required for all 
class one-configuration changes. As stated previously, IPTs are required to 
incorporate into the IPB the following: 

Functional and physical characteristics of the Configuration Item, 
any joint and combined operations that the configuration item, 
functional and physical characteristics used for production 
acceptance testing, and tests that are required for the deployment, 
installation, support, training, and disposal of the Configuration 
Item. (Defense Acquisition University (DAU), 2010) 

Section (c) provided an overview of HSI activities in the 
Engineering and Manufacturing Development phase. See Table 4 for a summary 
of HSI activities during the E&MD phase. Section (d) will provide an overview of 
the same issues for the Production and Deployment phase. 
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Table 4. E&MD Phase HSI Activity Summary 


HSI Activity 

Publication 

IDAT&L Activity 

1. Monitor activities to reduce risk. 

DoDI 5000.02, DAG (2010) 
and Shattuck (2009) 

Production and 

Manufacturing activities 

2. Review processes for HSI 
emphasis for MDA updates. 

DoDI 5000.02, DAG (2010) 
and Shattuck (2009) 

Acquisition Strategy and 

APB 

3. Provide HSI board members for 
RFP reviews. 

DoDI 5000.02, DAG (2010) 
and Shattuck (2009) 

Source Selection Evaluation 
Board (SSEB) 

4. Perform Modeling and Simulation 
activities: human performance 
modeling; mitigation of risk and 
safety issues. 

DoDI 5000.02, DAG (2010) 
and Shattuck (2009) 

Prototype Development 

5. Provide Manpower, Training, 
Personnel, HFE and Habitability 
updates due to mission capability, 
organization or funding change. 

DoDI 5000.02, DAG (2010) 
and Shattuck (2009); 
service policies 

Product Support Plan (PSP) 
and the Performance-Based 
Logistics (PBL) Strategy 

6. Identify Training requirement; 
Optimize Manpower/Personnel 
readiness requirements; Identify 

MPT Total Ownership Cost (TOC); 
Describe ESOH requirements; 
Describe Habitability Requirements. 

OPNAVINST 5310.23, Air 
Force 711"^ HSI 

Development Guide; Army 
MANPRINT documents 

Capability Production 
Document (CPD) 

7. Provide HSI input to ensure the 
designs are functional from the 
user’s perspective with minimal risk 
to them. 

OPNAVINST 5310.23, Air 
Force 711"^ HSI 

Development Guide; Army 
MANPRINT documents 

System Engineering 

Reviews: Test Readiness 
Review (TRR), the 

Production Readiness 

Review (PRR), the System 
Verification Review (SVR), 
and the Functional 
Configuration Audit (FCA). 

8. Participate in an IPT to provide 
functional and physical 
characteristics for the configuration 
Items. 

OPNAVINST 5310.23, Air 
Force 711"^ HSI 

Development Guide; Army 
MANPRINT documents 

Technical Outputs: IPB; 

Same as TD phase 


d. Production and Deployment Phase 

The overall focus of the Production and Deployment phase is to 
‘complete the development of a system or increment of capability, 
leveraging design considerations; complete full system integration; 
develop an affordable and executable manufacturing processes, 
complete system fabrication, test and evaluation’. (DAU Online, 
2010 ) 


The purpose of the Production and Deployment (P&D) phase is to 
achieve an operational capability that satisfies the mission need. This phase 
consists of two efforts: Low Rate Initial Production (LRIP) and Full Rate 
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Production and Deployment (FRP&D). The phase begins after a successful 
Milestone C review. The Full Rate Production Decision Review (FRPDR) 
separates the two efforts. MSI practitioners assist in these efforts by participating 
in integrated product teams and providing updates to documentation. 

Three reviews take place at the beginning of the Production and 
Deployment phase. These reviews include the Operational Test Readiness 
Review (OTRR), the Assessment of Operational Test Readiness (AOTR), and 
the Physical Configuration Audit (PCA). The OTRR is required prior to Initial 
Operational Test and Evaluation (lOT&E), which assesses the operational 
capabilities, suitability and effectiveness of the product. The AOTR occurs upon 
demonstration of those operational capabilities. The PCA determines if: 

Design documentation matches the item as specified in the 
contract; determines that the manufacturing processes, quality 
control system, the measurement and test equipment as well as 
training programs are planned, tracked, and controlled; and defines 
the starting point for controlling the detail design and establishing a 
product baseline. (Shattuck, 2009) 

MSI participation in these reviews is essential. Each review employs a checklist 
that is used to evaluate the systems. The checklists include activities such as 
Program Management, Logistics, Testing, Interoperability, etc. The OTRR 
checklist, for example, contains 96 FISI evaluation activities (out of a total of 143 
items). The PCA contains 143 FISI items. 

MSI is also present in the Defense Acquisition Program Support 
(DAPS) Methodology for System and Software Engineering. The methodology 
provides examples of MSI domain activities and provides criteria to evaluate the 
activities. Although the reference is not used extensively in this thesis to identify 
the MSI activities for the cost estimation process, it should be considered for 
future research. A DAPS example of the manpower domain guidance is as 
follows: 
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Manpower factors include job tasks, operation and maintenance 
(O&M) rates, workload, and operational conditions used to 
determine the number and mix of military and DoD civilian 
manpower needed to operate, support, maintain and provide 
training for the system. 

DoDD 5000.1 requires the Component Services to plan programs 
within projected future year manpower availability. Program 
manpower requirements should be based on studies and analyses 
that consider all operational facets of the Concept of Operations 
(CONORS) to account for the manpower mix, and the impact on 
any established Service-level constraints on manpower end 
strength. (Department of Defense, 2000a) 

Other HSI activities in the P&D phase include updating the Product 
Support Plans (PSP) and verifying Warfighter requirements in the Pre-Initial 
Operational Capability (IOC) review. As stated previously, HSI practitioners are 
not directly involved in writing the PSP but work with logisticians to update HSI 
criteria (requirements and performance metrics use to optimize the system) 
within the document. The IOC review updates configuration control changes, 
notes deficiency correction statuses within the system, and defines the maturity 
of the system. 

Two of the most important aspects of the P&D phase are the Initial 
Operational Test and Evaluation (lOT&E) and the Live Fire Test and Evaluation 
(LFT&E) activities. HSI practitioners (as part of IPTs) help to identify testable HSI 
metrics for these tests. OT&E activities occur prior to Low-Rate Production 
decisions. The OT&E ensures that suitability and effectiveness meets the 
requirements of the “Warfighter, government civilians, and DoD contractors that 
will operate, maintain, and support the system, and those who will train them” 
(Shattuck, 2009). The LFT&E tests for the vulnerability or lethality of the system. 
HSI is indirectly involved in this testing; however, practitioners can learn how the 
system design affects the susceptibility and recoverability of the system, thus 
mitigating safety and mishap concerns. 
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Finally, scheduling or funding cuts may remove HSI requirements. 
Therefore, prior to the FRP, the Acquisition Strategy, Acquisition Plan, Source 
Selection Plan, RFPs and Final Product Baseline documents receive FISI criteria 
updates. This is the last line of defense in advocating for the representative users 
of the product. These documents, as well as manpower estimations, economic 
analyses. Critical Safety List items, PESHE and TEMP, will be reviewed at the 
FRP by the MDA. 

Section (d) provided an overview of HSI activities in the Production 
and Deployment phase. See Table 5 for a summary of HSI activities during the 
P&D phase. Section (e) will provide an overview of the same issues for the 
Operation and Support phase. 


Table 5. P&D Phase HSI Activity Summary 


HSI Activity 

Publication 

IDAT&L Activity 

1. Provide updates to system 
documentation; Review system for 

HSI input. Program Management, 
Logistics, Testing, Interoperability 
system evaluation processes using 
DAPS methods. 

Shattuck, 2009; DoDD 
5000.1 

Operational Test Readiness 
Review (OTRR), the 
Assessment of Operational 
Test Readiness (AOTR), and 
the Physical Configuration 
Audit (PCA) 

2. Verify Warfighter requirements; 

3. Update PSP HSI criteria 
(requirements and performance 
metrics use to optimize the system) 
within the document. 

Shattuck, 2009 

Product Support Plans (PSP) 
and Pre-Initial Operational 
Capability (IOC) review 

4. HSI (as part of IPTs) help to identify 
testable HSI metrics. 

5. OT&E: Ensure that suitability and 
effectiveness meets the requirements 

6. LFT&E: Lessons learned from 
system design affects the 
susceptibility and recoverability of the 
system. 

Shattuck, 2009 

Initial Operational Test and 
Evaluation (lOT&E) and the 
Live Fire Test and Evaluation 
(LFT&E) 

7. HSI criteria updates. 

Shattuck, 2009 

Acquisition Strategy, 
Acquisition Plan, Source 
Selection Plan, RFPs and 

Final Product Baseline; 
manpower estimations, 
economic analyses. Critical 
Safety List items, PESHE 
and TEMP 
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e. Operation and Support Phase 

Life-cycle sustainment planning and execution seamlessly span a 
system's entire life cycle, from Materiel Solution Analysis to 
disposal. It translates force provider capability and performance 
requirements into tailored product support to achieve specified and 
evolving life-cycle product support availability, reliability, and 
affordability parameters. (DAU Online, 2010) 

The purpose of the Operations and Support (O&S) phase is to 
“execute a support program that meets materiel readiness and operational 
support performance requirements and sustains the system in the most cost- 
effective manner over its total life cycle” (DAU Online, 2010). The O&S phase 
fulfills this purpose with two efforts: Life Cycle Sustainment and Disposal. HSI 
practitioners assist in supportability assessments, conduct modeling and 
simulation (M&S) testing, and provide updates to documentation. The 
subsequent paragraphs describe HSI assistance during sustainment and 
disposal. 

During sustainment, the program manager and logisticians are 
concerned with the operational readiness and safety of the system. To maintain 
the readiness and safety of the system, supportability assessments and M&S 
activities take place to determine possible system improvements. HSI 
practitioners can assist in these processes by contributing to the supportability 
analysis, and supporting modeling and simulation studies. During these M&S 
studies, sustainability and mission effectiveness can be assessed. Lessons 
learned during operational field-testing are used for modifying subsequent 
system increments. Because of these changes, each increment requires an 
update to the ODD, CPD, and Key Performance Parameter (KPP) documents. 

Follow-On Test and Evaluation is performed when a system (or 
product) is tested in an operational environment. During this phase of testing, 
verification of the following suitability effectiveness aspects occurs: 
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Availability, compatibility, transportability, interoperability, reliability, 
wartime usage rates, maintainability, safety, human factors, 
manpower supportability, logistics supportability, and training 
requirements, environmental conditions of use scenarios (i.e., 
weather and climatic conditions, terrain effects, battlefield 
disturbances, and enemy threat conditions), mission length 
performance, vulnerability to countermeasures, and susceptibility of 
a system. (Shattuck, 2009) 

Disposal is the final process of any defense acquisition program. 
Disposal costs are a small portion of the system’s total life cycle cost; however, 
cost decisions occur prior to Milestone B (See Figure 8). The Defense Acquisition 
Guidebook (2010) states that program managers must determine the cost 
associated with “identification of plans for minimization and/or a safe disposal of 
hazardous materials, wastes, and pollutants associated with the system, and a 
compliance schedule for National Environmental Policy Act (NEPA)” (DAU, 2010, 
p. 283). Although indirectly associated, HSI practitioners (i.e.. Environmental, 
Safety and Occupational Health) provide input to help determine these costs. HSI 
practitioners ensure that environmental precautions (i.e., hazardous material 
exposure, safety measures and risk mitigation) that will affect the user are 
included in programs. 
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Figure 8. Weapon System Timeline and Life Cycle Cost Categories (From 

Mislick, 2009) 


Finally, the technical review process that occurs during the O&S 
phase is the In-Service Review (ISR). The ISR “ensures that the system is 
operationally employed; system risk is well-understood, properly managed, and 
designed to describe and characterize the health of the system and provides 
assessments that are measureable for risk, readiness, technical status and, 
trends” (Shattuck, 2009). Although the Systems Safety Working Group and the 
Integrated Logistics Management Team conduct the ISR, it must receive 
contributions from HSI practitioners to ensure the system is performing as 
specified. Table 6 summarizes the FISI activities during the O&S phase. 
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Table 6. O&S Phase HSI Activity Summary 


HSI Activity 

Publication 

IDAT&L Activity 

1. Maintain the readiness and safety of the 
system, supportability assessments and M&S 
activities take piace to determine possibie 
system improvements; 

2. Assess sustainabiiity and mission 
effectiveness. 

DAG, 2010, 

Shattuck, 2009 

Sustainment supportability 

3. Assist in providing updates due to system 
increment improvements. 

DAG, 2010, 

CDD, CPD, and Key 
Performance Parameter 
(KPP) documents 

4. Assist in verification of the following 
suitability effectiveness 

DAG, 2010, 

Follow-On Test and 
Evaluation 

5. HSI practitioners (i.e., Environmental, 

Safety and Occupational Health) provide 
input for environmental precautions. 

DAG, 2010, 

Disposal 

6. HSI practitioners provide input to ensure 
the system is performing as specified. 

Shattuck, 2009 

In-Service Review (ISR). 


Sections (a) through (e) have identified HSI activities from Concept 
identification to Disposal. It is now possible to create a more accurate picture of 
where HSI activity costs can occur as well as possible drivers that can contribute 
to the cost estimation processes. The next section will provide an overview of the 
cost estimation techniques, methods and processes used to develop a new HSI 
cost estimation process. 

B. COST ESTIMATION TECHNIQUES, METHODS AND PROCESSES 

Cost estimating, as part of a total systems analysis, provides an 
analytic underpinning to support decision makers. (Nussbaum, 

2010 ) 

This section discusses the basic steps of cost estimation. The topics 
include Cost estimation purpose, scope, ground rules and assumptions; 
Modeling and identification of cost drivers, and cost estimation methodology. 
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1 . 


Purpose, Scope, Ground Rules and Assumptions 


In any cost estimation activity, there are several issues that need to be 
determined at the beginning of the process. These issues include the purpose, 
scope, ground rules, and assumptions for the cost estimation process about to 
be undertaken. In keeping with this approach, this section makes explicit each of 
these issues. 


a. Purpose 

The purpose of the present effort is to refine current cost estimation 
processes that include HSI activities that take place during the entire acquisition 
life cycle. The refined processes will be used during the AoA as well as during 
other activities such as the program office estimate and the baseline cost 
estimates. 


b. Scope 

The cost estimation process will attempt to address all HSI 
activities throughout the entire acquisition life cycle. The process is intended to 
be generic in nature so that it can be applied to any military service and to any 
type of system. However, the research effort will emphasize issues such as 
development/operation test and evaluation processes, program increments, 
operation and support, and environmental costs. The cost estimation process 
should also include risk and cost relationships. 

c. Ground Rules 

The present effort is constrained by manpower (this thesis is an 
individual effort by one graduate student), by time (this thesis had to be 
completed within nine months) and by availability of other resources such as cost 
data and guidance, subject-matter experts, and funding for travel. 
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d. Assumptions 


The work reported herein is only an initial effort; additional work will 
be required to refine the cost estimation process for HSI activities. The process is 
based upon the data that were collected from subject-matter experts who were 
available. Although the current research is based predominantly on peacetime 
operations, future efforts should concern war-related cost. 

2. Identification and Modeiing of Cost Drivers 

The life-cycle phase review described earlier identified HSI activities that 
are essential to the AoA cost estimate process. In performing cost estimates, HSI 
driver trade-offs must also be determined. These estimates include the activities 
that help determine the projected cost of all HSI domains. However, emphasis is 
placed on manpower, training, personnel, and human factors engineering. This 
section discusses these cost drivers, cost estimate relationships, and modeling 
efforts. 


a. Manpower Cost Drivers 

Manpower cost drivers and relationships are the most commonly 
used and accepted variables in cost estimate methodology. The publications that 
discuss Manpower cost drivers and relationships include the Directive-Type 
Memorandum (DTM) 09-007 and the 2010 Federal Acquisition Regulations. 

The DTM-09-007 states that manpower costs are identified as 
“specific unit, organization, function, mission, or defense acquisition program... 
analysts should report the full costs of both military and civilian DoD manpower” 
(DoD, 2010a, p. 5). Cost drivers such as man-hour costs, are incorporated into 
the O&S phase of the life cycle. Although it is imperative to include the proper 
workforce mixture (military, civilian and contractor workloads), the highest 
performance at minimum cost is needed. Therefore, AoA processes include 
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estimates for comparison between these entities. These costs do not usually 
occur once, but span over the life of the system. For example, at any given point 
in time, the cost of a military member will include: 

• Base pay, 

• Basic substance and housing allowance, 

• Cost of medical care, and 

• Retirement 

The contractor cost may be short-lived. Contractor costs usually include 
development or short-term services involving testing, logistics, transportation, 
and will not include added benefits. Defining manpower business rules and 
methodology is the next step for defining manpower cost estimation 
relationships. 

The DTM-09-007 defines business rules and methodology for the 
workforce mix (DoD, 2010a). In order to determine appropriate manpower costs, 
composite rates for Manpower must be used. The Office of Undersecretary of 
Defense website provides these rates. Appendix A of this thesis also contains the 
current fiscal year composite rates. Furthermore, determining manpower cost 
includes describing direct and indirect costs. 

Direct costs that are often overlooked include the deferred 
compensation costs (liabilities/retirement or pay-as-you-go cost) that DoD 
civilians may incur (p. 7). Many tests and evaluations during development invoke 
hearing conservation programs. Occupational Health costs incorporate these 
programs and include hearing test costs or disability costs after the fact. The 
DTM-09-007 states the “full costs of manpower include current and deferred 
compensation costs paid in cash and in-kind, as well as non-compensation 
costs” (DoD, 2010a, p. 7). Short-term costs increase when workforce duration 
lengthens. These costs are associated with activities such as establishing or 
modifying day care and commissary services for military locations. Table 7 
includes an overview of the direct costs incurred. 
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Table 7. Direct Labor Cost Elements for Military and DoD Civilian Personnel (From 

DTM-09-007: DoD, [2010]) 



MRarv 

Civiian 

DoD 

Other Federal AoeiKy 

DoD 

Other Federal Agency 

Varable Costs in 
Short Rut 

Basic pay 

Alowances and special pa^ 

Incertvepa^ 

Heath benefit active duty and dependents 

Social Seciftty and htedicare 

Retiied pay(accnjai| 

TrareiFCStansportation subsidy 

Education assistance 

Heath benefit rebree (>d5 MERHCF acenjal) 
Traini ng costs l amortiaed over years of practice) 
Recnitmert advertising, etc. (amortized | 

Concurert receipt (Treasury) 
Mlitary Rebremert (Treasury) 
h€RHCF (Tieasuyl 

Child educabon(Eduoabcn) 

Basic payhocality pay 

Atowaraes and special pays 
hcertvePerformance awards 

Health benefit (government share of FEbBPl 

Social Secuityand htedicare 

Rebred pay (government share) 
Tra«lPCS^ansportabcn subsidyfreiccabon bonus 
Education assistance 

C>ieibme.holidayro(her pays 

Life insurance.Vwiiiei's compensabon benefits 
Recnjbng, advertising, etc. (amorbzed) 


Fled Costs in 
Short Ru\ 

Child develcpment 

Family sippcrt services 

Discout grocehes 


Child devetopmeit 


Deferred 

PayAs-Yoi><3o 

Costs 

Heath benefit rebree (<86 rebree and family) 
Heath benefit olher(TAMPandCHCBP) 
Discoul grocehes, rebree 

Separabon pay and have! 

Unemptoymeri benefits 

Death gratubes 

Siwvof benefits 

VA benefits (Veterans Affairs) 
Emptoymert training (Labor) 

Seveience health benefit 

Severencepayfincenbve 

Rebrement benefit (CSRS uAnJed) 
Health benefit 

Life insuance benefit 


The 2010 Federal Acquisition Regulation section 31.203 defines 
indirect costs. Indirect costs are those costs that are “not directly identified with a 
single final cost objective, but identified with two or more final cost objectives or 
with at least one intermediate cost objective” (Department of Defense [DoD], U.S. 
General Services Administration [GSA], National Aeronautics and Space 
Administration [NASA], 2010). Direct costs can be treated as indirect costs if the 
dollar amount is minor and the accounting treatment is applied consistently. 
These cost must however, produce results that are the same as if treated as a 
direct cost (DoD et al., 2010). These costs include machinery, facilities, materials 
and supplies; however, they vary from contract to contract. 

Frequently, DoD organizations write contracts for FISI services; 
therefore, evaluating indirect cost is essential. Unlike the work completed by DoD 
or civilians, many of the liabilities for contracting are not under sovereign 

immunity and must be factored in to the cost equation. If contract negotiations 
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leave out goods, benefits, and services for government-furnished property and 
cost of upper-level management, contracts will most likely increase in cost over 
time. 

The DTM-09-007 states that these overlooked contract costs can 

include: 

• Liability to third parties; 

• Reimbursements to a contractor for payments the contractor, its 
insurance company, or the Department of Labor make pursuant to 
Public Laws 85-608 and 77-784 (References (i) and (j)); 

• Reimbursements the Department of Defense is obligated to make 
with respect to publicly sponsored insurance (e.g., air carrier 
insurance sponsored by the Federal Aviation Administration 
pursuant to title 49, U.S.C. (Reference (k)) and;..., 

• The value of the authorization and consent to infringe privately held 
patents with freedom from monetary damages and injunctive relief 
that would otherwise prevent a contractor from performing the 
function. (DoD, 2010, p. 12) 

As with any system, analyzing the operational tempo and mission 
requirements will provide other influential cost drivers. Johnson, Osborn, Previc, 
and Prevost (2005) paper provides an excellent example of influential cost 
drivers. 

The authors addressed drivers by performing a top down 
requirements analysis (TDRA). This approach determined that the highest 
manning occurred during operational tempos under battle readiness and wartime 
amphibious assault operations. These conditions lead to increased watch 
standing and manning requirements. By including these types of influential 
drivers in the estimate, the impact of such outcomes as the need for equipment 
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redesign, lower than expected performance, and delays in schedule can be 
determined. These types of drivers are essential in producing reliable estimates 
for HSI related work. 

b. Personnel and Training Cost Drivers 

Personnel and Training domains also provide important cost drivers 
and relationships. Each grade, rank, rate, general service position or manpower 
contract type will have specific associated costs and are; therefore, factored into 
the estimate for O&S phase. Determining the personnel requirements and 
recruiting efforts for supplying the needed operators and maintainors of the 
system is another personnel cost. 

The training costs involved include developing and updating 
training strategy and plans, as well as developing effective training methods and 
tools. To produce qualified maintainors and operators for the various training 
pipelines, these methods and tools are essential. Training drivers include the 
technologies used as well as the test and evaluation of technology. Drivers 
should also consider the manpower cost to deliver the instruction. Therefore, 
both the R&D and O&S life-cycle phases consider training costs. 

c. Human Factors Engineering (HFE) Cost Drivers 

The HFE domain dominates Research and Development (R&D) 
cost estimates. The HFE cost drivers in R&D are essentially the cost of 
development, testing and evaluation. Developing tools, analyzing tasks and 
evaluating human-technology interactions are typical activities for HFE 
practitioners. These activities can account for hundreds and even thousands of 
man-hours contribute to these methods and research processes and thus should 
qualify as a high driver. 

Modeling and Simulation (M&S) costs occur in practically every HSI 
domain activity (i.e., manpower, training, safety and survivability) in a acquisition 
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program. For example, HFE costs drivers that relate to manpower include 
function allocation, workforce role determination, workload reduction, as well as 
the inherent workload risk and affordability determinations. Therefore, M&S is a 
high driver depending on the cost of technology, length of service, and the needs 
determination for modification of design. Use of analogous systems and 
comparable tools help determine the cost estimations. 

d. Environment and Safety Cost Drivers 

As stated previously. Environment and Safety have their own 
process owners. Periodically, practitioners must review and update safety and 
environmental inspection procedures, design requirements, and regulations. 
Besides licensure and environmental fees, a majority of safety costs are 
comprised of man hours required to perform needed administration and 
procedural requirements. The Environmental, Safety and Health Cost Analysis 
guide (1998) define two drivers of environmental cost: 

• Public Law 103-337, Section 815, Environmental Consequence 
Analysis of Major Defense Acquisition Programs, and 

• Department of Defense Regulation 5000.2-R, Mandatory 
Procedures for Major Defense Acquisition Programs (MDAPs) and 
Major Automated information System (MAIS) Acquisition Programs, 
Section 4.3.7. (p. 9) 

These programs discuss “materials to be used, the mode of operations and 
maintenance, requirements for demilitarization, and methods of disposal, after 
consideration of all pollution prevention opportunities and in light of all 
environmental mitigation measures to which the Department expressly commits” 
(Air Force Materiels Command, 1998). 

The Environment and Safety practitioners work hand in hand with 
HFE practitioners when designing the system for safe and effective handling, 
operation, and disposal. These are mainly manpower intensive drivers; however 
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when a system has a lifespan that exceeds initial life expectancy, the cost of 
reinstallation, deployment, upgrades and disposal will skyrocket. An example of a 
system that has gone beyond the normal lifespan and has required significant 
modification is the Iowa class battleships. Commissioned in 1942, the Iowa class 
ships received numerous modifications. Defense and survivability modifications 
included electronic warfare suites, close-in weapon systems, BGM-109 
Tomahawk Land Attack Missiles, and RGM-84 Harpoon anti-ship missiles. After 
48 years of service, the ships were finally decommissioned. 

e. Occupational Health, Habitability and Survivability Cost 
Drivers 

The final HSI domains of Occupational Health, Habitability and 
Survivability are man-hour intensive. These domains contribute cost drivers 
indirectly to the cost estimation. They provide cost variables for the R&D phase 
of the life cycle. Bratt, Doganiero, and Spencer (1998) discuses these human 
error and mishap related variables. 

As stated previously, indirect manpower costs include deferred 
compensation (e.g. disability). Bratt, et al. (1998) suggests that it is important to 
substantiate the driver causal factors; otherwise, consequences may be 
undetermined for any system development. Mishap data is relevant to any 
platform whether it is aircraft, ship or land based (including peace and war 
operations). The leading causes of mishaps are human error and equipment 
failure. 

Human error may occur from someone having a bad day, a crisis in 
their personal life, delayed or incorrect decision-making, or even unfamiliar or 
faulty written procedures. In many cases, the equipment failure also stems from 
human error (e.g. errors in design considerations etc.). Manpower reduction, 
poor or inadequate human factors engineering, and lack of attention to safety 
considerations are among the factors that contribute to these efforts. The 
potentially enormous price incurred for these errors translates into cost such as: 
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• disposal costs, 

• medical care, 

• disability and rehabilitation, 

• redesign and modification, and 

• training 

Identifying and mitigating the health risks involved in system development is a 
cost avoidance measure. Although not normally considered in a cost estimate, 
these potential outcomes certainly should be included. Human error is 
unavoidable, though it can be reduced. Cost estimates must include mitigation of 
these risks and mishaps (apparent in any system) and be compensated for when 
they occur. 


f. Cost Modeling 

HSI cost modeling can be adapted from current models in 
Occupational Health and System Engineering. These modeling activities are 
dependent on the system and target audience description for the intended use of 
the system. This section discusses these various models used in cost estimation 
of Occupational Health and Systems Engineering. 

Bratt et al. (1998) reports on a 1997 Army weapon hazard study 
that linked possible health hazards to clinic services. Specifically, the authors 
summarized their work as follows: 

Incidence rates and calculated costs based on industry-wide data 
on injuries, lost time, hospitalization, and disability, and this 
framework provides a method to reasonably estimate the medical 
and lost military manpower costs of unabated health hazards 
associated with Army materiel, (p. 443) 

The article addressed the effectiveness of the health risk assessment and 
management system while also contributing to the control of life cycle cost. The 
article provides the severity levels and hazard probabilities needed for cost 
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estimations (see Figure 9, Tables 8 and 9). The resource also includes valuable 
equations for factoring hazard variables into the cost estimations. A full variable 
list and explanations are available in Appendix B. 



Figure 9. Cost Components from a Single Hazard (From “Estimating the 

health hazard cost,” 1998) 


Table 8. Risk Assessment Codes (RAC) and Costs (Thousands of Dollars) Matrix 

(from “Estimating the health hazard cost,” 1998) 


Hazard Severity (SJ 

Hazard Probability (PJ 

Frequent 

Probable 

Occasional 

Remote 

Improbable 

A (.9) 

B(.5) 

C (.2) 

D (.01) 

E (.001) 

I (1) 

Catastrophic 

1 (SI 5,088) 

1 (S8,471) 

1 ($3,508) 

2 ($365) 

3 ($216) 

II (.1) 

Critical 

1 (SI ,410) 

1 ($783) 

2 ($313) 

3 ($16) 

4 (SI) 

III (.01) 

Marginal 

2 ($137) 

3 ($76) 

3 ($30) 

4 ($2) 

5 ($0,152) 

IV (.001) 

Negligible 

3 ($13) 

5 ($7) 

5 ($3) 

5 ($0,148) 

5 ($0,015) 


Notes: The calculations are based on a high risk system. The numbers 1,2, 3, 4, and 5, in the columns under 
Hazard Probability are the RACs. The numbers in parentheses in the columns under Hazard Probability are 
the medical costs that are incurred for a given RAC if no intervention occurs. 
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Table 9. Cost Component Equations (From “Estimating the health hazard cost,” 

1998) 


Re4ate<J Cost 
Component 

Component 

calculation 

Ec|uatBon 

All (except 
death costs) 

Number of people 
exposed to hazard 

N. = P.xN,x 

Clinic co<sts 

Number tyf people 
injured or 31 

N, = N. X S, X 1. 

Clinic costs 

Clinic costs 

C. = N. X 

Clinic costs 

Number of cfinic visits 

N, = N. X S,. X [V. {1, X N^)] 

H ospital ization 
costs 

Hospitalization costs 

X 

Hospitalization 

costs 

Number of persons 
hospitalized 

= N. X S. X 1, 

H ospital ization 
costs 

Number of hospital days 

N, = N. xS»xl„xS(D«xD,J 

Lost time costs 

Lost tinr«e costs 

C, = N, X W„ X B, 

Lost time costs 

Number of persons 
losing time 

= N. X S. X 1, 

Lost time costs 

Number of lost 
workdays 

N, = N. xS,xI„x5:(D„x DJ 

Disability costs 

Disab3ity costs 

X X 1, X T, X Z (D^ X BJ x 12 mo^ 

+ [(I, X B,) + (Ip X Bp)] 

Disability costs 

Number- of persons 
disabled 

Np.-N.xS.xCT.xl, -l.-lp) 

Rehabilitation 

costs 

Rehabilitation costs 

C, = N. X S. X l.x T, X X X Q,x B^ 

Rehabilitation 

costs 

Number of rehabilitation 

cases 

N, = N. X S. X I, X T, X XD, X Q, 

Death costs 

Death costs 

= (N« X B^ 

Note: Number of poopio exposed to hazard (N, — P»x Ns x N^ is a common term rotated to al cost compo¬ 
nents except death costs. 


Since the System Engineering community undoubtedly influences 
the HSI community, Valerdi (2005) provides further development of possible cost 
estimation relationships that are fundamental to HSI-derived models. 

Valerdi identifies system engineering effort (in person-months) 
using the Constructive Systems Engineering Cost Model (COSYSMO). This 
model includes four primary drivers (see Table 9), as well as fourteen effort 
multipliers/cost drivers (See Table 10). 
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Table 10. Size Drivers and Corresponding Data Items (From “The Constructive 

Systems Engineering Cost Model,” 2005) 


Dl■i^■el■ Name 

Data Item 

# of System Requirements 

Counted from the system specification 

# of Major Interfaces 

Counted fioin interface control document(s) 

# of Critical Alaoritluns 

Counted horn system spec or mode description docs 

# of Operational Scenarios 

Coiuited fiom test cases or use cases 


Table 11. Cost Drivers and Corresponding Data Items (from “The Constructive 

Systems Engineering Cost Model,” 2005) 


Driver Name 

Data Item 

Requirements imderstandmg 

Subjective assessment of tlie s>’stem requirements 

Architecture understanding 

Subjective assessment of the system architecture 

Level of service requirements 

Subjective difficulty of satisfymg the key performance 
parameters 

Migration complexity 

hifluence of legacy system (if applicable) 

Teclmology risk 

Matuiiry. readiness, and obsolescence of technology 

Documentation to match hfe cycle 
needs 

Breadth and depth of required documentation 

# and Diversity of 

Sites, mstallations. operatmg environment, and diverse 

mstalla tions/platforms 

platforms 

# of Recursive levels m the design 

Number of apphcable levels of the Work Breakdotvn Stmcture 

Stakeholder team cohesion 

Subjective assessment of all stakeholders 

PersonneL'team capability 

Subjective assessment of the team’s intellectual capabihtv 

Personnel experience'continuity 

Subjective assessment of staff consistency 

Process capabihty 

CMMI level or equivalent rating 

Multisite coordination 

Location of stakeholders and coordmation barriers 

Tool support 

Subjective assessment of SE tools 


Valerdi initially expresses System Engineering effort in terms of 
historical data, size drivers and effort multipliers. The final derived formula for 
system engineering effort, however, includes weighted variables for the effort 
multipliers (See Figures 10 and 11). These equations identify the relationships 
between the effort multipliers and the cost factor variables. Each factor is subject 
to the changes in system use such as “redesign, reimplementation, and retest” 
(Valerdi, 2005, p. 41). Factors are also impacted by complexity and volatility. The 
model encompasses periodic adjustment for the drivers. Furthermore, Valerdi 
states that increases or decreases to variables are system or criteria dependant. 
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Equation 2 


PM= .4 - iSizeY ■ n EM, 

7=1 


W'liere: 

PXIxs = effort ill Person J^Iontlis CNoniinal Sclieclule) 

A. = calibration constant derived from historical project data 

Size = determined by computina the weighted stun of the foiu size dii\'ers 

E = represents economy/diseconomy of scale; default is 1.0 

n = number of cost diivers (1-1) 

E>I,' = effort multiplier for the /u, cost driver. Nominal is 1.0. Adjacent 
multipliers ha\e constant ratios (aeometric progression). W'^ithin theii' 
respective rating scale, the calibrated sensitivity range of a multiplier is the 
ratio of liighest to lowest value. 


Figure 10. COSYSMO Operational Form (From “The Constructive Systems 

Engineering Cost Model,” 2005) 



Equation 6 


Figure 11. Final COSYSMO CER (From “The Constructive Systems 

Engineering Cost Model,” 2005) 

In Figure 11, the symbol expressions include: 


k = number of requirements, interfaces, algorithms, and operational 
scenarios (REC, INTF, ALG, OPSC) 
i/v= weight 
e = easy 
n = nominal 
d = difficult 
O = driver count 
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Next, Valerdi establishes categories for the COSYSMO model 
factors (See Figure 12). In HSI, identity of such factors is necessary due to the 
role of the HSI practitioner in policy writing, development, testing, modeling and 
simulation of systems. 


UNDERSTANDING FACTORS 

- Requirements understanding 

- Architecture understanding 

- Stakeholder team cohesion 

- Personnel experience/continuity 

COMPLEXITY FACTORS 


PEOPLE FACTORS 

- Personnel/team capability 

- Process capability 

ENVIRONMENT FACTORS 

- Multisite coordination 

- Tool support 


- Level of service requirements 

- Technology Risk 

- # of Recursive Levels in the Design 

- Documentation Match to Life Cycle Needs 


OPERATIONS FACTORS 

- # and Diversity of Installations/Platforms 

- Migration complexity 


Figure 12. Cost Factor Clustering (From “The Constructive Systems 

Engineering Cost Model,” 2005) 


Finally, Figure 13 and Table 12 depict Valerdi’s final cost model 
(and the acronym listing). The COSYSMO model provides the foundation that 
HSI models can build upon. Future HSI modifications should enhance this 
current model. 


Equation 14 

log[SE_HRS_ADJ] = logfSIZH] + loa^CUNDERSTANDrSTG] -i- 

loaCCOlMPLHXITY] — lo 2 :[OPHR.A.TIONS] -i- 
logfPEOPLE] + loa[H>J\*IROT<rMENT] 

\\4iere 

IJl^TDERSTANOrSTG = REQU * ARCH * TEAM * PEXP 
COMPLEXITY = LSVC * TRSKl * RECU * DOCU 
OPERATIONS = INST * MIGR 
PEOPLE = PCAP * PROC 
EN\TRONMENT = SITE * TOOL 


Figure 13. COSYSMO Model (From “The Constructive Systems Engineering 

Cost Model,” 2005) 
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Table 12. COSYSMO Predictor Descriptions (From “The Constructive Systems 

Engineering Cost Model,” 2005) 


Predictor 

Term 

Description 

s, 

log(REQ) 

# of System Requirements 

S2 

log(INTF) 

# of Major Interfaces 

S3 

log(ALG) 

# of Critical Algorithms 

S3 

log(OPSC) 

# of Operational Scenarios 

EM, 

log(RQMT) 

Requirements Understanding 

EMj 

log(ARCH) 

Architecture Understanding 

EM3 

log(LSVC) 

Level of Service Requirements 

EM3 

log(MIGR) 

Migration Complexity 

EMj 

log(TRSK) 

Technology Risk 

EMf 

log(DOCU) 

Documentation to match lifecycle needs 

EM7 

log(INST) 

# and diversity of installations/platforms 

EM. 

log(RECU) 

# of recursive levels in the design 

EM, 

log(TEAM) 

Stakeholder Team Cohesion 

EM,o 

log(PCAP) 

Personnel/Team Capability 

EM„ 

log(PEXP) 

Personnel Experience/Continuity 

EM,2 

log(PROC) 

Process Capability 

EM ,3 

log(SITE) 

Multisite Coordination 

EM ,4 

log (TOOL) 

Tool Support 


3. Cost Estimation Methodoiogy 

Cost estimators typically use four methodologies. Each methodology is 
suited for different periods in the life cycle although they may be used in any of 
the life cycle periods. These methods are; engineering build-up (bottom-up 
technique); analogy; parametric; and expert opinion. This section discusses 
these methods and describe the manner in which they are used. 

Acquisition programs frequently employ a Work Breakdown Structure 
(WBS) or a Cost Analysis Requirement Descriptions (CARD). When a detailed 
WBS (at the lowest level) is available for the system of interest, the engineering 
build-up method is the preferred cost estimation method. When other programs 
of a similar nature (complexity, technical or physical) are available, the analogy 
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method is preferred. In general, whenever few actual costs are available early a 
program, similar programs are the most viable means for determining cost 
estimates. 

The parametric method is preferred if the program is in the early stages of 
development and limited data (program and technical) are available. This method 
is similar to the analogy method. It uses similar system performance and design 
characteristics while determining cost element relationships (CER) on which to 
determine statistical inference. If these methods are not available, then it is 
possible to use expert opinion. It is also possible to use a combination of these 
methods, if the system allows. Reproducibility and sensitivity analysis are 
techniques used to improve the accuracy of the cost estimate. Figure 14 provides 
a comparison of these methods’ advantages and disadvantages 


Model 

Category 

Description 

Advantages 

Limitations 

Analogj" 

Compare project 
with past sirmlar 
projects 

Estimates are based 
on actual 
experience 

Tmly similar 
projects must exist 

Expert 

Judgment 

Coasult with one or 
more experts 

Little or no 
historical data is 
needed: good for 
new or unique 
projects 

Experts tend to be 
biased: knowledge 
level is sometimes 
questionable 

Bottoms-E'p 

Indrsiduals assess 
each component 
and then 
component 
estunates are 
summed to 
calculate the total 
estimate 

Accurate estimates 
are possible 
because of detailed 
basis of estimate 
(BOE); promotes 
indisidual 
responsibihty- 

Methods are time- 
consummg; detailed 
data may not be 
available, 

especially early in a 
program; 

integration costs are 

sometimes 

disregarded 

Parameti'ic 

Models 

Perform overall 
estimate using 
design parameters 
and mathematical 
algorithms 

Models are usually 
fast and easy to use. 
and useful early m 
a program; they are 
also objective and 
repeatable 

Models can be 
inaccurate if not 
properly calibrated 
and validated; it is 
possible that 
historical data used 
for cahbration may 
not be relevant to 
new programs 


Figure 14. Methodology Comparison (From Nussbaum, 2010) 
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Figure 15 provides recommendations for which method to use during 
various phases of the acquisition life cycle. 



Figure 15. The Cost Estimating Process: Estimating Approaches and Cost 
Methodologies and Milestones (From Nussbaum, 2010) 


A majority of MSI activities have not been defined in most system Work 
Breakdown Structures. Therefore, this thesis uses expert opinion to determine 
MSI costs during the acquisition life cycle. However, the long-term goal is to gain 
enough HSI cost data in similar systems to use the parametric method. This 
method will ensure a higher level of accuracy based on the CER used. 

If a parametric method can be employed, CERs are determined based on 
the following formula: 

Cost = f (technical, performance, schedule) (Nussbaum, 2010) 

In doing so, it will be possible to use statistical techniques such as multivariate, 
bivariate, univariate analyses as well as log-linear analysis. The next step is to 
normalize the data in order to compare similar systems. 

As stated in Nussbaum (2010), data normalization reduces content 

differences such as “elements of cost; technology similarities, and consistency in 

scope of work”. However, the availability of historical HSI data limits data 
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normalization for cost estimation. When comparing similar systems, it is also 
important to break down activities into common cost elements. For example, an 
estimate in one program may contain a combination SE/PM cost element but 
historical documents list these elements separately. Division of cost elements 
reduces replication. Mapping the differences in the same manner in estimation 
mitigates the likelihood of over- or under-estimating. This will be apparent in cost 
estimates involving HSI criteria since many process owners (HSI domain criteria) 
are cost oriented this way. Since HSI elements are typically under current 
processes (i.e., manpower, training, etc.), this will inevitably lessen the task of 
creating new factors to explain costs. The best estimation methods rely on the 
analyst’s judgment and the data available. 

The current thesis investigates methods for performing estimates involving 
HSI criteria. The thesis is intended as a resource to guide the HSI modification of 
current cost models and estimation methods. 

C. HSI CONCEPT MODEL 

The purpose of this section is to describe two HSI models that will inform 
the HSI cost estimation process. The models include domain variables, 
constraints, enablers, and cost drivers. The two HSI models are the 711*^ Human 
Performance Wing’s HSI model, (23 July 2007) (see Figure 16) and the HSI 
Process Model developed by Miller and Shattuck (2007) (see Figure 17). 
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Figure 16. Linkages between the HSI process model and the Joint Capabilities 
Integration Development System gap analysis. (From “Human Systems 
Integration (HSI) in Education,” 2007) 


The Human Performance Wing HSI Model (see Figure 16) is described as 
follows: 

The four input domains [of HFE, Personnel, Training, and 
Manpower] greatly simplifies the challenges of forecasting the 
impact of HSI trade-offs through modeling and simulation, a 
necessary consideration given DOD initiatives for simulation-based 
acquisitions. In contrast, the ESOH, habitability, and survivability 
domains represent desired system attributes or behaviors not 
directly procurable; rather, they emerge through various 
combinations of the input domains. (Tvaryanas, Brown, & Miller, 

2009) 

Although the article provides an adequate assessment of the constraints 
and enablers of a system, it does not represent the cost variables associated 
with performance optimization. However, the model developed by Miller & 

Shattuck (2007), could prove useful for estimating the cost of HSI activities. 
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Weighting factors or cost inputs (relevant to previous systems seen in the initial 
block of domain advocacy), initiate processes depicted in the model. The 
factors/inputs that could augment those already listed in the Domain Advocacy 
block include Manpower Estimates, Training Technology Development costs. 
Modeling and Simulation (M&S), and Test and Evaluation (T&E) costs. The 
constraints and enabling factors of the model include schedule delays, contractor 
renegotiations, performance testing, and development failures. These factors 
interact with the inputs to optimize the system outcome. 


System Acquisition Lifecycie Environment Inciuding 

System Requirements and Capabiiities 


As constrained or 




enabled by 

Yields/Assessed/ 



Development Cost 

Evaluated by 


Domain 

Delivery Schedule 



Advocacy for 

Design Risk 

1 

Usability 

Safety Statistics 


M + P + T + HFE + Sur + 


Mishap Rates 

Total 

Hab + Sys Saf + H Haz 





burvivaDiiity " 

Habitability 

Performance 

(This advocacy can take 
various forms, e.g., 
standards, formulas. 

(The process is 
influenced by 
these contextual 

Maintainability 

Reliability 

Operating and Lifecycle 
Costs 


models, regulations. 

variables.) 


expertise.) 




(Inputs) 


(Enablers and (Intermediary or 
Constraints) First Order 

Outcomes) 


(Final 

Outcome/ 

Product) 


Figure 17. HSI Process Model (From Miller & Shattuck, 2007) 


Miller and Shattuck (2007), state as “factors in the first two boxes 
combine, they produce effects that can be captured by outcome measures such 
as usability, survivability, habitability, maintainability, reliability, safety (indicated 
by safety statistics and mishap rates), operating and total life cycle costs” (p. 12). 
The outcome measures are further identified by cost high drivers such as 
deferred compensation, disability cost and cost avoidance and system 
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engineering factors (i.e., mean time between failure, maintenance hours etc.). 
The product of all equations will be the total system performance of the human- 
system product. The Miller and Shattuck model is a more functional model (in 
terms of cost estimations) than the 711 th model. 

Most expert cost estimators agree, “simpler is better,” and that the use of 
sophisticated tools and methods of cost estimation comes with its own burdens. 
Many advanced methods utilize “black box” applications in which the user has 
little knowledge of the underlying algorithms or processes. Novice estimators 
must often use of these complicated cost estimation tools (e.g., ACEIT, SEER-H, 
SEER-S, PRICE-H, PRICE-S, etc.) until experience is gained. This thesis 
employs a relatively simple Microsoft ® Access based database to identify HSI 
activities and their associated cost. Excel-based operations perform the 
necessary activity calculations. This thesis will use the HSI process model by 
Miller and Shattuck (2007). However, the thesis will leverage the processes 
embedded in cost estimation tools such as the Automated Cost Estimating 
Integration Tool (ACEIT) and the Marine Corp Ground Training Cost estimation 
tool. 

Section C reviewed HSI Concept models for future cost estimation 
processes. Section D will review current HSI cost estimation methods used by 
other countries and organizations. 

D. CURRENT SERVICE HSI COST ESTIMATION METHODS 

Reports by Canadian Defense Technology Center (DTC) (2006); Brooks, 
Greenley, Dyck, Salwaycott, Scipione and Shaw (2008); and Liu (2009) provide 
guidance for determining a method to perform HSI cost estimation. The 
Canadian Ministry of Defense initiated the DTC HFI program in 2003, which is 
equivalent to U.S. HSI. 

DTC is a formal collaborative arrangement between industry and 
academic experts in a particular technology, funded jointly by 
participants and the MOD. The participants work together to 
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generate and enhance the technology vital to the delivery of future 
UK Defence capabilities. (British Aerospace Systems, 2010) 

The Canadian Defense Technology Center Report (2006) identifies five 
arguments including the cost considerations of five categories related to Human 
Factors Integration (HFI) in the overall cost estimate of a system (see Table 13). 
The purpose of this study was to make decisions on budget allocation regarding 
HFI activities. Due to a misunderstanding of the value of HFI and the inability to 
trace intangible benefits, previous work had not provided compelling evidence for 
including HFI into cost estimations. The arguments lay a foundation of cost 
factors and provide the rationale for incorporating HFI; however, the bases of 
these cost factors are sometimes vague or obscure. 
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Table 13. Human Factor Integration Arguments 


Argument 1 

HFI can reduce major costs. 

Argument 2 

HFI is increasingly required due to developments in 
technology. 

Argument 3 

HFI plays a critical role in identifying and mitigating 
operational risks. 

Argument 4 

HFI can draw on resources that enable an effective and 
efficient process. 

Argument 5 

Effective HFI requires early, complete and close involvement. 


The report identifies the value of HFI contributions by providing successful 
(and non-successful) case studies related to HFI initiatives (or the absence of 
initiatives). The analyses performed on these case studies show the savings or 
expenditures related to specific HFI variables. The five Arguments and their 
related case studies are now described. 

Argument 1 includes operational and developmental risk cost. Main costs 
include: 

• Accident costs (system safety or human error), 

• Lowered performance and mission failure, people and materiel 
resources, 

• Inefficient development and implementation processes (Technology 
and Development phase; Integration phase of life cycle), 

• Redesign cost (integrate phase), and 

• Personnel costs (Procurement-training and recruitment) 

Typical case studies within this category include aviation mishaps, 
equipment designs (rifles), unanticipated cost due to extra training, improper 
implementation and late design changes. In some cases, it is hard to determine 
the actual cost related to HFI. For example, the author incorporates full cost of 
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machinery such as plane loss, but omits identification of hazards. This problem 
occurs when reporting cause of failure. It is important to perform accurate 
tradeoff analysis and identification of hazards for costs/health hazards 
relationships. 

Argument 2 relates to complexity driven errors. Increased complexities in 
technology, mandated standards and/or regulatory actions support the argument 
for HFI. Tradeoff integration should provide insight into complexity related errors; 
however, the costs often do not consider these tradeoffs. 

Argument 3 determines the need for identifying and mitigating operational 
risk. This occurs through raising awareness, identifying areas of uncertainty and 
providing a means for user input. User-guided input leads to a better design with 
less rework. A valid methodology (through modeling of potential problems) is the 
key to cost savings. Early identification of standards and substantiated 
requirements that can be used later in scenario development also saves cost. 

Argument 4 indicates that HFI practitioners can utilize previous 
methodologies, principles and research to prevent repeating past mistakes. The 
report establishes a need for documenting these processes to mitigate future 
Human Factors problems. 

Finally, Argument 5 states the need for cost estimation throughout the 
entire process of the life cycle. Four methodologies for cost estimation include 
Investigating (problems and constraints). Creating (solutions based on 
requirements). Evaluating (AoA), and Managing (resources, organization, etc.). 
Furthermore, the argument states the need for immersing HFI activities early in 
the process. This will allow practitioners to develop and maintain an efficient and 
cost effective solution. 

The Defense Technology Centre report provides excellent examples of 
HFI problems and their associated costs. The report also provides provisional 
insight into requirements, validity and reliability, increased efficiency and the 
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need for user involvement. However, future work should include detailed analysis 
of cost factors and their linkages to HFI domains across the life cycle. 

Brooks et al. (2008) provides a second HSI cost process. The Canadian 
Department of National Defence (DND), in partnership with Defense Research 
and Development of Canada, provided a multi-year analysis of activities to 
demonstrate and validate an HSI approach for the Canadian Materiel Acquisition 
and Support community of DND. 

From 2001 to 2004 an analysis of 31 HSI-related acquisition projects were 
completed and then reviewed for cost measures. The study produced measures 
in terms of immediate and extrapolated savings based on the HSI activities in 
these projects. The study also provided examples of future budget considerations 
such as incorporating cost estimation ranges. The purpose of the report was to 
establish the need for HSI. The report included the following topics: concept of 
HSI, development of HSI programs, team implementation, integration of HSI 
process, tools of HSI, case studies (evaluation, implementation, support, 
definition of tools and lessons learned) and cost tracking data for the application 
of HSI. The most relevant aspects of the report with respect to this thesis are the 
tool applications and the cost tracking data. The subsequent paragraphs 
summarize these aspects. 

It is important to note that in the Canadian DND, HSI consists of five 
domains. In the U.S., the number of domains varies from one military service to 
another (U.S. Air Force, nine; U.S. Navy, eight; U.S. Army, seven domains). The 
Canadian DND combines the manpower and personnel domains and the system 
safety (SS) and health hazards (HHA) domains. The SS-HHA combination is 
equivalent to the U.S. domains of system safety, occupational health, 
survivability, and environment. The integration of the domains allowed for more 
centralized control over projects and better integration of HSI activities to scale. 
Comparable to the U.S. approach to HSI, the DND also uses MIL HBK 46855A 
(HFE), MIL STD 882D (SS). In addition, the DND uses CFITES (the standard for 
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training development). In order to perform cost estimations for U.S. acquisition 
programs using Canadian data, these composite domains and their cost data will 
have to be decomposed. 

The study identified three categories of savings in the 31 case studies. 
These categories can be applied to nearly all HSI cost estimation activities. 
These categories included immediate savings, extrapolated savings and 
uncalculated savings. These three measures provide a useful taxonomy that can 
be employed across the life cycle of any program. The article described 
immediate savings as items or resources that saved time and money during the 
work process. The extrapolated savings included those applications of HSI 
processes that occurred early in the life cycle, which led to decisions that saved 
money over time. The third, uncalculated savings, results from the impact of 
applying HSI rather than saving lives or improving operational effectiveness. 

The methodical implementation of HSI activities within the Canadian DND 
involves several documents and activities, including: an HSI plan. Concept of 
Operations and Concept Support (CONSUP); Target Audience Descriptions 
(TAD); System Design Inputs; and System Evaluations. Human Factors 
Engineering (HFE) domain activities begin with a task analysis, and then mission, 
function and task decompositions (backbone of system safety assessments). 
Next, workload analysis and predictive studies are use to support the personnel 
domain requirements. Third, the results of the detailed task analysis are used to 
support training needs assessments, HHA assessments and SS assessments. 
Forth, the process focuses on trade-off analysis. 

The results of the analysis in the report indicated that these projects used 
that HSI practices included 200 of the 400 modeling and simulation tools that 
were available. These tools included Human Form Computer Aided Drafting and 
Design (CADD) tools, prototyping, simulation tools, task network modeling (for 
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workload analysis), and mission tools, as well as function and task analysis 
(MFTA) tools. Case studies focused on five areas of analysis. The Brooks et al. 
(2008) areas included: 

• HSI program development: HSI program. Directorate of Technical 
Airworthiness support (DTA), Modeling and Simulation 
Coordination office definition, HSI Concept of Operations, TTCP 
HSI workshop 

• Major HSI processes: Joint Intelligence information Capability 
(JIIFC), Advanced Land Fire Control System (ALFCS), Future 
Armored Vehicle System (FAVS), Multi Mission Effects Vehicle 
(MMEV), Maritime Helicopter Project, MHP modeling and workload. 
Very Short Range Air Defense (VSHORAD) 

• HSI Process subset: Visual Acuity for divers. Grasshopper 
Uninhabited Aerial Vehicle (UAV); Canadian Patrol Frigate (CPF) 
Accommodation, Advanced Linked Extended Reconnaissance and 
Targeting (ALERT) Experimental Design Support) 

• HSI tool evaluation: Helmet Mounted Display (HMD) for CF18, 
Clothe the Mounted Soldier (CMS) survey. Medium Logistics 
Vehicle Wheeled (MLVW) survey. Surveillance, Target Acquisition, 
Night Observation (STANO) survey and Collaborative Displays. 

Provisions of HSI and project definition support: CB plus Program and 
performance projection framework, Cipro Plus requirements. Collaborative 
Planning and Management Environment (CPME), Directorate of Training and 
Education Programs, 3D modeling. Nuclear Biological Chemical Defense 
(NBCD) Repository Protection Program, Project Activity Reporting System 
(PARS) (Brooks, 2008). 

Through pre-determined criteria, a comparison of case studies produced a 
likely range of budget percentages (4-20% HSI cost of the project engineering 
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budget). An example of the budgetary considerations included the designing of 
“next generation” interfaces for fire control systems and experimental programs. 
These activities provided impact assessment for crew sizes and organizational 
structure. Developmental programs applied HSI costs, but they also considered 
to Commercial-off the Shelf (COTS) products. Humans Factor and Safety will 
impact COTS and system integration and will also affect the installation of the 
system. Program considerations were not the only concern. The subsequent 
paragraphs describe the reports discussion of the role of cost benefit analysis. 

The report tracked the application of HSI cost benefits to determine a cost- 
benefit framework for the studies. The analysis included only calculable costs 
and savings. Of the five categories listed previously, only categories 2 and 3 
were considered for the cost-benefit calculations. The other categories did not 
include immediate or extrapolated saving. 

An example of a cost-benefit calculation included payback amounts 
produced by HSI applications. By removing unnecessary or redundant 
functionality early in the design process, significant savings can occur. For 
example, the removal of redundant displays resulted in $5,000,000 worth of 
savings (Brooks et al., 2010). If the payback or savings is substantial, the 
omission is a success. Uncalculated savings may occur when design changes 
positively impact the quality of life or the life expectancy of the Warfighter. These 
savings also impact safety and enable effective operations (Greenley & 
Associates, 2008). However, these costs were not used in a total cost-benefit 
analysis because of their multi-variant, circular and discountable characteristics. 
Future analysis should include these costs. The report suggests that if 
uncalculated savings can be estimated, there is the likelihood the cost savings 
would be in the range of millions of dollars. 

The report follows the case study analysis with the methodology required 
to demonstrate the cost of HSI. The data produced budgetary percentages for 
HSI allocations. The percentages of HSI investment were consistent across 
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projects. This result was regardless of the size of the program. The changes 
were a function of the HSI activities required by the program (i.e., human 
interface project complexity initiated increases in engineering budgets). 

In conclusion, the DND report provides an alternative cost estimation 
methodology. It attempts to demonstrate a method for integrating the various 
domains of HSI. This is important because no process is a stand-alone process. 
Trade-off analysis must occur across the domains to correctly determine all 
estimation variables. While the report provides excellent information on 
immediate and extrapolated savings, it also mentions that uncalculated savings 
need to be analyzed more specifically which will result in exponential savings. 

Liu (2009) explains the need for HSI cost measurement by examining 
Total Ownership Cost reduction in Development and Procurement stages of the 
life cycle. The programs of choice for this analysis were the Unmanned Aerial 
Systems (UAS). HSI, System Engineering and Program Manager costs were 
minimal compared to the one to five billion dollar RDT&E, O&M, Military 
Personnel (MILPERS) and Procurement budgets for the UAS programs 
examined. Liu described the current estimation method as a “Rule of Thumb” 
measure. This approach is based on the number of man-hours and the total 
engineering development. In the “Rule of Thumb” procedure, historical systems 
data influence the estimate. Factors that influence the cost estimate include 
expert opinion, technology changes, aircraft weights and number of units. The 
approach used in the article was parametric estimation methodology. This type of 
methodology incorporates size drivers such as number of requirements, 
interfaces, scenarios, algorithms and volatility factors. The method also uses 
effort multipliers (application factors, team factors, and schedule drivers). 
Together, the insertion of size drivers and effort multipliers into the Constructive 
Systems Engineering Cost Model (COSYSMO) modeling tool result in the total 
System Engineering effort. 
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To implement the model, the requirements are decomposed. The 
requirements decomposition should address the following; 

• Determining the system of interest, 

• Determining whether the requirements can be tested, verified or 
designed, 

• Describing the relationship of the system of interest to the total 
system, and 

• Assessing the number and the complexity of the requirements 

Along with the requirements, the effort multipliers are established. There 
are 14 effort multipliers. These multipliers include issues such as understanding 
requirements and architecture, migration complexity, personnel, tool support, and 
many others. High-level integrated process teams provide these multipliers. 

The article demonstrates the use of the model on HSI domain 
requirements that were pre-determined in government furnished documents. The 
author uses the COSYSMO model to describe national budget percentages for 
HSI activities. The author also provides suggested required inputs 
(variables/requirements specified earlier in this thesis), identifies useful outputs 
(identification of cost drivers, risk issues, and discrepancy warnings early in the 
program), and describes the model’s application to pre-Milestone A related 
activities. The Liu (2009) article provides a better understanding of requirement 
impacts and an alternative cost estimation approach that appears to be more 
viable than the traditional methodology. 

E. SUMMARY 

In summary, sections (A) through (D) discussed the decision processes, 
concept models, and methodologies that are essential to formulate a new HSI 
cost estimation approach. HSI and cost estimation practitioners can use this 
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approach for HSI activities throughout the acquisition life cycle and AoA. The 
following paragraph summarizes these sections. 

Section (A) discussed the decision support system and the Life Cycle 
Management System HSI activities. This section addressed the HSI activities 
(and associated cost drivers) that occur across the acquisition life cycle. Section 
(B) investigated the basic cost estimation methods for performing estimates 
involving HSI criteria. The section discussed the Bratt et al. (1998) and Valerdi 
(2005) cost methodologies that emphasized HSI criteria adaptations of current 
cost methods in Occupational Health and System Engineering. These methods 
provided a resource to guide the HSI modification of current cost models and 
estimation methods. Section (C) described the Miller and Shattuck (2007) and 
the U.S. Air Force 711*^^ Human Performance Wing (2006) HSI Models that will 
inform the new HSI cost estimation process developed in this thesis. Section (D) 
discussed reports by Canadian Defense Technology Center (2006), Brooks, 
et.al, (2008), and Liu (2009). These reports provide useful insights for 
determining a new method that will improve the cost estimation of HSI activities 
throughout the acquisition life cycle. 

The literature review emphasized the need to identify calculated and 
uncalculated savings, direct and indirect costs, benefits, risks, and trade-offs as 
well as cost variables that are of particular interest to HSI cost estimation. These 
sections combined, provided the necessary background to develop the HSI 
activity database and cost models presented in the next section of this thesis. 

An ultimate goal of HSI cost estimation will be the ability to perform 
parametric cost estimates with HSI historical costs (that are not currently 
available). Parametric cost methods, that use approved legacy system WBS, 
would then be treated as baselines that could then be incorporated into the 
estimates for funded requirements. The development of an HSI activity database 
is necessary to perform this type of estimate. 
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The next chapter introduces the first iteration of the proposed HSI Cost 
estimation process through development and verification of the HSI database 
and cost model. The chapter discusses the steps taken in developing the HSI 
activity database, followed by the initial SME Review verification studies. Next, 
the development of the HSI Cost model is discussed. Finally, the subsequent 
chapters discuss the recommendations for future verification, validation, and 
sensitivity analysis studies. 
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III. METHOD 


A. METHOD OVERVIEW 

This thesis uses an exploratory research method to identify those HSI 
activities in the acquisition life cycle that should be considered when performing 
AoA cost estimations. A database of these activities provides a framework for 
future historical cost estimation processes. The study elicits Subject-matter 
experts (SMEs) feedback to determine the accuracy and supportability of the HSI 
activities in each of the life cycle phases. The feedback is then used to modify 
the HSI activity database for future validation studies, as well as to promote 
better cost estimation processes. Furthermore, the current HSI cost estimation 
methods are compared in terms of the variables they consider. 

1. Participants 

The Navy HSI Working Group and the Air Force HSI Office, in partnership 
with the Air Force 711**^ Human Performance Wing, supplied the SMEs for this 
study. The two pre-selected groups of SMEs were assigned a number (one 
through four) that corresponded to a life cycle phase. The assignments 
accounted for SME core competency areas when available (e.g.. Requirements). 
Because the SMEs were participants to the research and were not subjects of 
the research, the Naval Postgraduate School’s Institutional Review Board (IRB) 
determined this study was not human subjects research. 

The activity review was designed to elicit many different viewpoints (i.e., 
holistic, domain specific). SMEs were identified as experts in HSI, an HSI 
domain, and/or in cost estimation. The study included 25 U.S. Air Force and 16 
U.S. Navy respondents. The respondents’ communities of practice and 
competency areas for the review were as follows (see Figures 18 and 19). The 
Navy respondents consisted of one military officer, 12 civil service employees. 
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and 3 government contractors; the Air Force respondents consisted of one 
military officer, 15 civil service employees, and nine government contractors. 
Three Navy respondents provided cost estimation expertise. However, Air Force 
cost estimation experts were unavailable during the review period. 

U.S. Navy SME 
Communities of Practice 


11 ..^ 

N»v»l $«« Cost Ottur SpiKO und ContrJKtor 

Systems Estimation ttaval v'.-’arfare 

Command Systems 

ComiMand 


Figure 18. U.S. Navy Communities of Practice 

U.S. Air Force SME 
Communities of Practice 
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Figure 19. U.S. Air Force Communities of Practice 
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B. SME REVIEW MATERIALS 


1. Database Development 

A significant part of this thesis involved creating a database of HSI 
activities. The activities were extracted from research articles, government 
policies, and subject-matter experts. Comparison of existing cost estimation 
methods provided an additional source of HSI activities and cost drivers. After 
compilation of these activities, an initial database was developed (Database is 
available in NPS Electronic Storage). 

The database to be use during the SME Review was designed as a 
Microsoft ©Access (HSI activity) database that was exported using a Microsoft © 
Excel file format (see Figure 20). 
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Figure 20. HSI Activity Access Database (excerpt) 
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SME Review Design 


The SME review consisted of an Excel workbook containing three 
spreadsheets (i.e., Column Heading and Descriptions, Phase Activities, and Cost 
Drivers) (see Appendix C). The column description spreadsheet provided a 
proposed list of database column headings that were considered important to the 
cost estimation process (see Figure 21). 
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Figure 21. SME Review HSI Database Column Descriptions (excerpt) 

The activity spreadsheet included 98 (major) life-cycle phase activities 
(MSA-24; TD-23; E&MD-22; P&D-17; 0&S-12) and their corresponding sub¬ 
activities (see Figure 22). Additionally, the spreadsheet included potential HSI 
domains and cost drivers that affected each major activity. 
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Figure 22. SME Review HSI Activity Spreadsheet (excerpt) 


85 


































The final spreadsheet contained a generic cost driver list (see Figure 23). 


Cost Drivers 

Cognitive Workload 

Physical Workload 

Complexity 

Ratforms) 

ClicalityofTask 

Fhocess capability (CMMI level or equivalent rating) 

Difficulty of Task 

Requirements understanding (Subjective assessment of the system requirements) 

Documentation to match life cycle needs (Breadth and depth of required documentation) 

Size 

Frequency of Task 

SLOC(Software lines of code) 

Initial Training (Instructor & Key Ffersonnel) 

Specialty Ffersonnel 

Initial Training (New Equipment at system fielding) 

Stakeholder team cohesion (Subjective assessment of all stakeholders) 

Level of service requirements (Subjective difficulty of satisfying the key performance 
F^rameters) 

Target Audience 

Maintenance Concept 

Technology risk (Maturity, readiness, and obsolescence of technology) 

Maintenance Requirement 

Time Constraint for System Development 

Mbnning 

Tool support (Subjective assessment of SEtools) 

Migration complexity (Influence of legacy system (if applicable)) 

Testing Type-detailed component testing 


Testing Type-subsystem integration 

Modeling & Simulation 

Testing Type-system simulation and stimulation 

Multisite coordination (Location of stakeholders and coordination barriers) 

Testing Type-campaign or force-level (system of systems). 

Occupational Training 

Type 

Operational Capability 

Unit/Sustainment Field Training 

Ffersonnel experience/continuity (Subjective assessment of staff consistency) 

Warranties 

FfersonneLteam capability (Subjective assessment of the team's intellectual capability) 



Figure 23. SME Review Cost Driver List 


Finally, a workbook was assembled that contained the Column Heading 
and Cost Driver spreadsheets, as well as one of the following phases: Materiel 
Solution Analysis, Technology and Development, Engineering & Manufacturing 
and Development, and combined Production and Deployment and Operation & 
Support. 

3. Historical Data Collection 

Historical system data also were collected from the Air Force Total 
Operational Cost (AFTOC) website. Defense Cost and Resource Center 
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(DCARC), and the ASN (RDA) website. These data were not included in the 
present study but could very well be used in future validation studies. The data 
included RDT&E product development, support, and test & evaluation costs, as 
well as O&S manpower, personnel, and training costs. The specific platforms for 
which data were available included the Predator (MQ-1), Global Hawk (RQ-4), 
and the Broad Area Maritime Surveillance Unmanned Aircraft System (BAMS). 

C. SME REVIEW PROCEDURES 

1. Distribution 

The activity database review was administered via e-mail to pre-selected 
SMEs. Each participant received an introductory e-mail explaining the purpose 
and description of the review materials. Next, an e-mail containing the review 
materials and an instruction sheet was sent (see Appendix D). Respondents 
were asked to complete the review on or before a pre-determined due date (21 
days later). Additionally, each respondent received a reminder e-mail midway 
through the review period. 

2. Procedures 

SMEs were required to review a selected (assigned) portion (a particular 
phase or phases) of the database and provide feedback. The review instructions 
were as follows. SMEs were required to review each spreadsheet to determine if 
the data were appropriate for AoA cost estimates. Next, SMEs were required to 
identify any data modifications (additions, deletions, corrections) that were 
required. Each spreadsheet provided blank rows for the SMEs to suggest 
missing activities. Furthermore, SMEs were asked to estimate the percentage of 
HSI man-hour effort required to complete the activity. SMEs were instructed that 
each of the phases was assigned a maximum value of 100% man-hour effort for 
a generic ACAT I program. Finally, SMEs were asked to provide an overall 
percentage of HSI man-hour effort for the entire phase. 
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Once the review was completed, the SMEs were instructed to save the 
document using an Excel (97-2003) format (“samefilename_yourinitials.xls” 
format) and to e-mail the results to the review administrator. 

3. Coding Process 

Table 14 illustrates the coding process implemented to classify database 
errors. Any data block that received a CR, INC or I NCR code by the review 
administrator were modified in the next database increment. 


Table 14. SME Review Codes 


Code 

Category 

Explanation 

AA 

Additional Activity 

SME added a recommended activity. 

C 

Correct 

No (or) minimal correction needed (less than 20%). 
Corrections include spelling errors, word changes or activity 
item changes. The Percentage basis is the total number of 
words or activity action items. 

CR 

Correct with 
Recommendation 

SME (or Review analyst) considers the activity correct and 
the SME provides a recommendation for activity adjustment 
(e.g.. Split, re-sequence, roll-up ‘include as second level to 
another activity’). 

E 

% Effort* 

SME provided man-hour effort percentage for HSI activity. 

GC 

General 

Comments 

SME provided a question or statement that did not allow 
analyst to apply another code. 

INC 

Incorrect 

Multiple corrections needed (greater than 20%). This 
percentage is based on number of words or activity items. 

INCR 

Incorrect with 
Recommendation 

SME (or Review analyst) considers the activity incorrect 
and the SME provides a recommendation for activity 
adjustment (e.g.. Split, re-sequence, roll-up ‘include as 
second level to another activity’). 

NC 

No Comment 

SME did not respond or the SME has considered the data 
appropriate. 

NQ 

Not qualified 

SME considered themselves inexperienced to comment on 
the activity. 

R 

Recommendation 

SME provide a recommendation for correction of the 
activity components. 


*Percent Effort was based on 100 % for each phase. 
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D. DEVELOPMENT OF HSI COST ESTIMATION MODEL 

The HSI Cost Estimation Model development is based on all of the HSI 
activities that could contribute to a new cost estimation process. The model also 
combines previously validated models, cost variables, and cost drivers. The 
review administrator applied an (S) code to similar method components and a 
(DS) code for dissimilar components. The (DS) coded components were 
recommended for further review. 

As stated during the review of the Valerdi’s COSYSMO model (2005), a 
cost estimation model should be a function of System Engineering man-hour 
effort (see Chapter II, Figure 11). The cost estimation model for this thesis is 
based on the Valerdi model, but is a function of HSI effort and contains 
variables/drivers that are elicited from other methods and case studies (e.g., 
Canadian Defense methods and other HSI domain methods). The following 
section provides the review results and the proposed model. 
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IV. SME REVIEW RESULTS 


The SME Review method provided response rates, adjustment 
percentages, driver types and HSI-Effort percentages. The following sections 
describe these results. 

1. Activity Review Results 

Table 15 reports the overall response rate by type of respondent and total 
contact count (the number of SMEs that received the review). 


Table 15. Total Response Rate 


Total Contacts 

83 

Total Resjwndents 

40 

Overall % Response 

48% 

total % Government Hesfonse 
(Military aid (5vil Service) 

70% 

Total % extractor Response 

30% 


Table 16 contains the response rates by service and phase. The response 
rates ranged from 14% to 63%. The most responses were received for the MSA 
phase (63%), while the fewest responses were received for the TD phase (14%). 
Navy rate of return percentages were higher overall, however, had a smaller 
sample size (USN=15, USAF=25). The P&D and O&S phases resulted in the 
same rates of return (USN=57%; USAF=46%). 
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Table 16. Descriptive Response Rates 


Phase 

MSA 

T4D 

E&MD 

P&D 

o&s 

Phase # 

1 

2 

3 

4 

c 

Service Gnouo 

US AF 

USN 

US AF 

USN 

USAF 

USN 

USAF 

USN 

US AF 

USN 

Total Phase Contacts 

14 

8 

14 

6 

15 

6 

13 

7 

13 

7 

Total Go vemm ent CGO V) 

8 

7 

10 

C 

10 

C 

8 

6 

8 

6 

Total Co ntractor(CTR) 

6 

1 

4 

1 

5 

1 

5 

1 

5 

1 

Total Respondents 

5 

5 

2 

1 

6 

1 

6 

4 

6 

4 

Total GOV 

3 

4 

2 

1 

3 

1 

4 

3 

4 

3 

Total CTR 

2 

1 

0 

0 

3 

0 

2 

1 

2 

1 

H Total Response 

36% 

63% 

14% 

17% 

40% 

17% 

46% 

57% 

46% 

57% 

% GOV Response 

38% 

57% 

20% 

20% 

30% 

20% 

50% 

50% 

50% 

50% 

% Contractor Response 

33% 

100% 

0% 

0% 

60% 

0% 

40% 

100% 

40% 

100% 


SME recommendations supported the accuracy of the database. Overall, 
the adjustments recommended ranged from 11% to 24% of the total number of 
entries for each phase. Table 17 provides a breakdown of the review phases, as 
well as the percentages of adjustments required. A data block is considered to be 
a single entry into the spreadsheet or database. The Total Activity Data Blocks 
column consists of the number of activities (i.e., the numbers in parentheses in 
the 1®* column) multiplied by six (the number of columns in the database). The 
Total Coded Responses consists of the Total Activity Responses multiplied by six 
(i.e., the number of columns in the database). 


Table 17. Database Descriptive Breakdown and Accuracy Rates 


Acquisition Phase 
(# Activities) 

Total Activity 
Responses 

Total Activity Data 
Blocks 

Total Coded 
Responses 

Total Adjustments 
Required 

Percentage 

Adjustment 

Required 

MSA (24) 

154 

144 

924 

183 

20% 

T&D (23) 

39 

138 

234 

57 

24% 

E&MD (22) 

82 

132 

492 

53 

11% 

P&D (17) 

60 

102 

360 

74 

21% 

0&S(12) 

27 

71 

162 

19 

12% 
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Table 18 provides a breakdown of the Total Coded Responses (from 
Table 17) by coding category and phase. Table 18 also provides additional detail 
with respect to the Total Adjustments Required column (in Table 17). The 
numbers in this column (of Table 18) are based on the following coding 
categories: Correct with Recommendation (CR); Incorrect (INC); Incorrect with 
Recommendation (INCR); or, General Comment (GC). The Percentage 
Adjustment Required (PAR) results consisted of dividing the Total Adjustments 
Required datum by the Total Coded Responses. 


Table 18. Activity Code Breakdown by Phase 



NC 

E 

C 

CR 

GC 

NQ 

INC 

INCR 

MSA 

464 

103 

167 

90 

42 

5 

2 

49 

T&D 

82 

46 

49 

32 

2 

0 

0 

23 

E&MD 

299 

44 

96 

25 

9 

0 

0 

19 

P&D 

163 

34 

89 

36 

26 

1 

0 

11 

O&S 

69 

23 

51 

6 

12 

0 

0 

1 

NC=No Comment 

E= Effort Given 

C=Correct 

CR= Correct w/Recommendation 


GC=General Comment or Question 
NQ=Not Qualified to comment 
INC=lncorrect 

INCR=lncorrect w/Recommendation 


2. Database Column Review Results 

SMEs were asked to review the database columns to determine whether 
they would provide the appropriate information necessary for establishing a 
historical cost database for HSI activities. Each SME reviewed 19 data blocks. 
Tables 19 and 20 provide the response rates and coding categories for this 
review (see Appendix C for a full listing of columns and descriptions). Overall, the 
database columns received positive reviews. SMEs considered all columns 
useful for establishing a historical database in support of HSI cost estimation 
activities. The data blocks that required adjustments (30%) were the based on 
SME general comments (GC) or recommendations (CR) for enhancements to the 
database. 
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Table 19. Database Column Response Rate 


Total Participants 

13 

Total Responses 

124 

Total Data Blocks 

19 

Total Coded Comment Blocks 

244 

Total Adjustment RQD 

72 

Percentage Adjustment RQD 

30% 


Table 20. Column Coding Categories 


C 

CR 

INCR 

INC 

NC 

NQ 

GC 

51 

43 

0 

0 

120 

1 

29 


3. Cost Driver Section Review 

The following list provides the results from the review of the cost drivers. 
SMEs commented on the accuracy of the proposed cost drivers and classified 
them by cost type (i.e., (D)irect or (l)ndirect). Overall, the list of cost drivers 
appeared to be adequate. However, some SMEs noted that certain drivers were 
specific to HSI activities while other drivers could be applied to many other 
acquisition activities. Respondents suggested that three drivers be removed 
(Size, Platform and Type). One of these drivers (type), however, was acceptable 
upon correction to the term system type. 
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Cost Driver Results List 


(I) Cognitive Workload 

(I) Complexity 

(I) Criticality of Task 

(I) Difficulty of Task 

(I) Frequency of Task 

(I) Level of service requirements 

(Subjective difficulty of satisfying 

the key performance 

Parameters) 

(I) Mission 

(I) Modeling & Simulation 
(I) Multisite coordination 
(Location of stakeholders and 
coordination barriers) 

(I) Personnel 

experience/continuity (Subjective 
assessment of staff consistency) 
(I) Personnel/team capability 
(Subjective assessment of the 
team’s intellectual capability) 

(I) Physical Workload 
(I) Process capability (CMMI 
level or equivalent rating) 

(I) Requirements understanding 
(Subjective assessment of the 
system requirements) 

(I) Stakeholder team cohesion 
(Subjective assessment of all 
stakeholders) 

(I) Target Audience 
(I) Technology risk (Maturity, 
readiness, and obsolescence of 
technology) 

(I) Time Constraint for System 
Development(l) Tool support 
(Subjective assessment of SE 
tools) 


(D) Documentation to match life 
cycle needs (Breadth and depth 
of required documentation) 

(D) Initial Training (Instructor & 
Key Personnel) 

(D) Initial Training (New 
Equipment at system fielding) 
(D) Maintenance Concept 
(D) Maintenance Requirement 
(D) Manning 
(D) Migration complexity 
(Influence of legacy system (if 
applicable)) 

(D) Occupational Training 
(D) Operational Capability 
(D) SLOC 

(D) Specialty Personnel 
(D) Testing Type-detailed 
component testing 
(D) Testing Type-subsystem 
integration 

(D) Testing Type-system 
simulation and stimulation 
(D) Testing Type-campaign or 
force-level (system of systems) 
(D) Unit/Sustainment Field 
Training 
(D) Warranties 
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In addition to the classification of the listed drivers, the SMEs provided 
additional drivers as seen in the following list. The list was further classified by 
the administrator to include cost type (i.e., direct (D) or indirect (I)) based on the 
Federal Accounting Regulations classification process. 


(D) Facilities - Development and 
test 

(D) Facilities: Operations and 

maintenance 

(D) Facilities: Training 

infrastructure 

(D) Force Structure Composition 
(Active, Reserve, Officer, 
Enlisted, Civilian, Contractor) 

(D) Safety requirements and 
regulations 

(D) Special tools & equipment: 
development; 

operations and maintenance 

(D) Specialized tools required for 

maintenance 

(D) Training location 

(I) Habitability shortfalls that lead 

to workforce attrition 

(I) Accessibility 

(I) Complexity (sub-categories): 

technological complexity & 

organizational complexity 

(I) Configuration and Change 

Management 

(I) Consequence of errors 

(I) CCTS and GCTS suitability 

assumptions 

(I) Errors identified 

(I) Facility layout 

(communication, distance to 

amenities, distance from noise 

source) 

(I) Function allocation (human, 
machine, supervisory, manual) 


(I) Intuitive usability (can novices 
operate without any training) 

(I) Material selection 

(I) Modeling & Simulation 

Validation 

(I) Poor Situation Awareness 

(I) Poor usability 

(I) Required KSAs 

(I) Requirements Determination 

(I) Situational awareness of 

automation state 

(I) Specialized safety 

(I) Specialty Personnel 

(development) 

(I) Team design 
(I) Test safety 

(I) Training Analysis and Design 
(I) Training pipeline (or other 
training "time" metric) 

(I) Under-manning 
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4. 


HSI Effort Results 


The final analysis consisted of looking at percentages of HSI-Effort across 
the life cycle phases for a general ACAT I program. The results in the following 
sections are intended to provide a starting point or baseline for HSI costs. The 
results in Tables 21 through 25 are ordered according to the acquisition cycle 
phases. A paper format precludes presentation of the entire database. Therefore, 
the tables show only the first two lines of each row. A complete database is 
available in electronic format upon request. Additionally, each table displays the 
total number of SME participants for each phase. Furthermore, the data 
presented in these tables is non-normalized. 

Table 21 provides the Materiel Solution Analysis Phase HSI-Effort results. 
Pre-Materiel Solution Analysis (or Pre-Materiel Development Decision) activities 
were provided an event coding of (A) through (F) while the Materiel Solution 
Analysis activities are coded (G) through (S). The results ranged an average of 
3% to 28% for the various activities. In other words, SMEs estimated these 
activities would represent between 3% and 28% of the total (i.e., 100%) HSI 
effort for the MSA phase. The highest HSI activity percentages were reported for 
the following activities: 

• Provide HSI activity input during Capabilities Based Assessment for 
ICD (M=23%, SD=0.20); 

• Provide an HSI assessment based on the best available solution 
descriptions (in CBA, AoA, Study Guidance, ICD) across all HSI 
domains and to define the risk level in each. (M=28%, SD=0.21); 

• Provide a total system analysis based on functional relationship 
perspectives with an emphasis on system boundaries (M=13%, 
SD=0.11); and, 

• Participate in the AoA assessment (M=10%, SD=0.02). 
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Furthermore, USN reviewers suggested combining the domains of Environment, 
Occupational Health and Safety while USAF reviewers replaced Occupational 
Health with Health Hazards. 


Table 21. MSA HSI-Effort Percentage (based on input from 10 SMEs) 


Event 

HSI_Activitv 

tiEAH% 

BlHt 

pucdon 

STTDV 

A. 

Provide HSt activity input during Capabilities Based 
Assessment forICD 

23% 

0.20 

B. 

Provide HSI input into the HSIP section o' the System 
Engineering Plan (SEP) 

8% 

0.03 

c. 

Provide a total system analysis based on functional 
relationship perspectives with an emphass on system 

13% 

0.11 

D 

Provide HSI considerations into tie AoA Study plan 

3% 

0.02 

E. 

Provide HSI input into ESOH Integration Strategy. 

6% 

0.04 

F. 

Perform HSI assessments based on best availabe solution 
description (in CBA. AoA, Study Guidance. ICD) across a 1 

28% 

0.21 

E. 

Perform an H SI analysis of legacy systems with respect to 
each domaih ard addressed to potential integrated impact 

8% 

0.04 

F 

Contribute to assessment and development of system 
concepts /Materiel solutions. 

4% 

0.01 

G. 

Assist in development of cost estimate besed on anticipated 
HFE, Manpower. Personnel. Traninq, safety activities/ 

4% 

0.01 

H. 

Conduct Mission Task Analysis. 

7% 

0.04 

1 . 

Conduct a Initia Top Down Reqcirements Analysis. 

8% 

0.02 

J. 

Assist in performance of Manpov/er /Personnel alternatives 
and concept development and assessments 

4% 

0.01 

K 

Identify Manning needs, manpower lessons learned, and high 
drivers from legacy systems. 

4% 

0.02 

L. 

Assist in linear modeling and studies analysis of system 
approach alternatives. 

7% 

0.02 

M. 

Conduct Manpower/Training assessments to determine 

KPPs learning cbjectives and training reguirements for 

6% 

0.04 

N. 

Assist in deriving systematic preciction of conditions through 
Initial Safety, and Occupational Health requirements and 

4% 

0.01 

O. 

Participate in AoA assessment 

10% 

0.02 

P, 

Define concept capabilit es and requirements 

3% 

0.02 

Q. 

Provide HSI input into Technology Development Strategy 

3% 

0.00 

R. 

Provide HSI Input into Life Cycle Sustainment Plan. 

4% 

0.02 

S. 

Provide HSI input into Support System Engineering 

Technical Reviews (SETRs) 

3% 

0.01 
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Tables 22 and 23 provide the HSI Effort for the Technology Development 
phase. For this phase, SMEs estimated these activities would represent between 
3% and 20% of the total (i.e., 100%) HSI effort. Several of the activities that are 
listed (callout/ guide or new activities suggested by SMEs) have only one SME 
effort contribution. These values have a standard deviation of zero. 

The highest HSI Effort percentages were attributed to the following activities: 

• Determine the need for an HSI IPT or working group (M=20%, 
SD=0.00), 

• Demonstrate & validate System & Technology maturity versus 
defined user needs & environmental constraints (M=20%, SD=0.00) 

• Decompose functional definitions into Critical Component 
Definitions & Technologies Verification Plan (M=16%, SD=0.00) 

• Demonstrate enabling/critical Technology components versus plan 
(M=13%, SD=0.08), 

• Perform analysis and recommendations for system technology 
development (M=12%, SD=0.12), 

• Provide HSI input into the Capability Development Document and 
Acquisition Baseline (M=11%, SD=0.08; M=11%, SD=0.08), and 

• Conduct formal development of human-centered source selection 
criteria for RFPs at Milestone (MS) A (M=11%, SD=0.13). 
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Table 22. Technology Development Phase HSI Effort 
(based on input from three SMEs) 


EVENT 

HSLActivity 

MEAN Activity 
Effort 

SD 

GUIDE 

Review all the inputs to the Technology Development Phase 
including the following: 

20% 

(one 

value) 

A. New 
Activity 

Determine the need for an HSI IPT or a working group 

20% 

(one 

value) 

B. 

Establish a HSI IPT or Working Group - consistent with the 
size and complexity of the envisioned acquisition program 

10% 

0.03 

GUIDE 

Throughout the next 5 steps C-G conduct trade offs and 
support the AoAs with the identified sub activities. 



C. 

Determine all relevant Critical Operational Issues and 
determine and develop Measures of Effectiveness 

5% 

0.01 

D. 

Callout 

Develop System Performance (&Constraints) Specification 
& Enabling/Critical Technologies & Prototypes Verification 

10% 

(one 

value) 

E. 

Callout 

Develop Functional Definitions for Enabling/Critical 
Technologies/Prototypes & Associated Verification Plan 

10% 

(one 

value) 

F. 

Callout 

Decompose functional definitions into Critical 

Component Definition & Technologies Verification Plan 

16% 

(one 

value) 

G. 

Identify HSI technical risks and mitigation strategies for 
technology development. 

10% 

(one 

value) 

GUIDE 

Throughout the next 7 Steps H-0 conduct trades that are 
spread across the identified sub activities. 



H. 

Demonstrate enabling/critical technology components 
versus plan 

13% 

0.08 

I. 

Provide integrated HSI assessment methodology for 
technology down selection recommendations and use in full 

7% 

0.05 

J. 

Refine and update requirements 

10% 

0.00 

K. 

Assist in technology selection 

7% 

0.05 

L. 

Perform analysis and recommendations for system 
technology development. 

12% 

0.12 

M. 

Assist in development of System Functional Specs & 
Verification Plan to evolve system functional baseline 

6% 

0.06 

N. 

Participation in Integrated Test Team (ITT) and Evaluation 
or other Integrated Performance Team IPT as may be 

8% 

0.03 
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Table 23. Technology Development Phase HSI Effort (continued) 


EVENT 

HSLActivity 

MEAN Activity 
Effort 

SD 

Guide 

Throughout the next 7 Steps P-S conduct trades that are 
spread across the identified sub activities. 

20% 

(one 

value) 

P. 

Callout 

Demonstrate enabling/critical technology components 
versus plan 

16% 

(one 

value) 

Q. 

Callout 

Demonstrate system & prototype functionality versus plan 

10% 

(one 

value) 

R. 

Callout 

Demonstrate/model integrated system versus performance 
specification 

10% 

(one 

value) 

S. 

Callout 

Demonstrate & validate system & technology maturity 
versus 

20% 

(one 

value) 

GUIDE 

Insure the production and completion of inputs to the 
following products: 



T. 

Review and provide input into Preliminary design and 
allocation baseline. 

8% 

0.10 

U. 

Provide HSI input into the Capability Development 

Document 

11% 

0.13 

V. 

Provide input into the Acquisition Strategy for goals, 
objectives and entrance and exit criteria as a member of 

3% 

0.03 

W. 

Provide input into Acquisition Program Baseline (APBs) 

11% 

0.13 

X. 

Incorporate requirement for contractor generated HSIP(s) 
and associated source selection criteria in Request for 

8% 

0.03 

Y. 

Conduct formal development of human-centered source 
selection criteria for RFPs at Milestone (MS) A 

11% 

0.13 

Z. 

Apply analysis in findings to Test and Evaluation Strategy 
(TES), LCMP and TEMP 

6% 

0.06 

AA. 

Provide input into System Engineering Review process 
(SRR). 

7% 

0.04 

BB 

Provide input into System Engineering Review process 
(SFR) 

10% 

(one 

value) 

CC 

Provide input into System Engineering Review process 
(PDR) 

16% 

(one 

value) 
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Table 24 provides the HSI Effort for the Engineering and Manufacturing 
Development Phase. The results for these activities ranged from 2% to 8% of the 
total HSI effort. Activities that resulted in the same percentage contributions (i.e., 
5%,5%, 5%) display a standard deviation of zero. The HSI activities with highest 
effort percentages included: 

• Assisting in development of test plans for integrated system testing 
(M=8%, SD=0.03); and, 

• Providing HSI input into technical reviews and Systems 
Engineering activities (M=8%, SD=0.04). 

SMEs suggested moving two activities from the MSA phase to the TD phase. 
These activities pertained to proving HSI input into the draft CDD and the 
Training Program Plan. 


Table 24. E&MD Phase HSI Effort (based on input from seven SMEs) 


Event 

Category 

HSI_Activity 

MEAN % Effort 

SD 

/K. 

PrcDN/ide HSI inpul: to the “TEfS/IP 

5% 

0.00 

B. 

Finalize HSI Rec| ui i re me n ts toy 
ototaining agreement from 

2 % 

0.01 

c . 

Assist in development of 
detailed design rec| lj i re me n ts 

2 % 

0.02 

D. 

Assess detailed system 
designs to ensure that they 

4 % 

0.02 

E . 

Participate in prototypes/test 
materials development (for 

5% 

0.04 

f=. 

Evaluate prototypes (or test 
materials) for HSI Criteria 

ISJ e w 
Activity 


G. 

Participate in Design 
Demonstrations (to include 


0.03 

H . 

Evaluate HSI criteria for the 
system during component / 

5% 

0.00 


Perform HSI evaluation on 
System Components 

4 % 

0.02 

J . 

Assist in Development of Test 
Plans for integrated system 


0.03 

K. 

Participate in HSI test events 


0.02 

I_. 

Participate in ECP 
development 

3 % 

0.02 

IVI. 

Provide HSI updates to 
Acquisition Strategy and 

2 % 

0.01 

ISI. 

Provide HSI input into 

Ca pa toi 1 ity P rod u ct io n 

4 % 

0.01 

O. 

Provide HSI input into 
technical reviews and Systems 

S % 

0.04 

R*. 

Provide HSI input for error and 
fault analysis 

3 % 

0.01 
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Table 25 provides the HSI Effort for the Production and Development 
phase. The events coded (A) through (N) ranged from an average of 4% to 10% 
of the total HSI effort. The highest HSI effort percentages were reported for the 
following activities: 

• Test and Evaluation support and participation (LFT&E) {M=^0%, 
SD=0.07); 

• Assisting with the System Hazard Analysis (SHA) and Operating 
and Support Hazard Analysis (M=8%, SD=0.04); (M=13%, 
SD=0.11); and, 

• Participate in the AoA Assessment {M=^0%, SD=0.02). 

SMEs suggested the insertion of three additional activities. 


Table 25. Production and Deployment Phase HSI Effort 
(based on input from 10 SMEs) 


Event 

HSI_Activity 

Mean Activity 
Effort 

(% Man Hours) 

SD 

A. 

HSI PIsnning snd 

5% 

0.01 

B. 

Review end Develop HFE 

T% 

0.04 

C. 

Verify HSI requirements for 

4% 

0.01 

D. 

Assist in T&E HSI snelyses of 
system design end instelletion 


0.02 

E. 

Assist in T&E HFE enelyses of 
system design end instelletion 

e% 

0.01 

F. 

T&E support end perticipetion 

e% 

0.03 

G. 

T&E support end perticipetion 
(LFT&E) 

1 o% 

o.or 

H. 

T&E support end perticipetion 
(FOT&E) 

r% 

0.02 

1 . 

Review OPTEVFOR reports 

5% 

0.00 

J. 

Document end enelyze post 
deployment lessons leerned 

5% 

0.01 

K. 

Provide input into Review 
process for P&D Phese 

4% 

0.02 

L. 

Updete Menpower/Personnel 
requirements for system 

e% 

0.01 

M. 

Updete Treining systems 
requirements for system 

e% 

0.00 

N. 

Assist with System Hezerd 
Anelysis (SHA)end Opereting 

8% 

0.04 


103 























Table 26 provides the HSI-Effort contributions for the Operation and 
Support phase. The results for activities (A) through (K) ranged from 5% to 15% 
of the total HSI effort. The highest HSI activity percentages reported for the 
activities included: 

• Provide Follow-on Test and Evaluation support {M=^5%, SD=0.07); 
and, 

• Provide testing and verification of Follow-on Test & Evaluation 
processes {M=^2%, SD=0.03). 

SMEs suggested incorporating two of the original database activities into 
activities that pertained to additional Manpower or HFE support during Full 
Operational Capability Support. SMEs did not remove any HSI activities from the 
database but did add one activity (HSI planning and management). 


Table 26. Operation and Support Phase HSI Effort (based on input from 10 SMEs) 


Event 

HSI_Activity 

MEAN % Effort 
(Man flours) 

SX DEV 
(% EFFORX) 

A. 

HSI Flanning and 

IVIanagennent 

8% 

New 

Activity 

B. 

Provide Follow-on Test and 
Evaluation support 

1 5% 

0.07 

C. 

Provide analysis for HFE FOC 
support and 

8% 

0.01 

D. 

Provide recommendations for 
HFE FOC support and 

8% 

0.03 

E. 

Provide Training FOC support 
and logistics/sustainment 

9% 

0.01 

F. 

Provide IVIanpower/Personnel 
FOC support and 

6% 

0.01 

G. 

Analyze and prototype post¬ 
product improvements for the 

7% 

0.02 

H. 

Provide input for sustainment 
of operational readiness and 

7% 

0.04 

i- 

Provide testing and verification 
of Follow-on T&E processes. 

1 2% 

0.03 

J. 

Provide FOC support and 
logistics/sustainment 

8% 

0.03 

K. 

Provide input into O&S review 
processes (ISR) 

5% 

0.01 
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5. Results Summary 

The SME Review provided results that included the Descriptive and Total 
Response Rates, Database Adjustment Rates, Cost Driver use and type 
determination and HSI-Effort. 

This thesis returned an overall response rate by service and type of 
respondent. The most of the responses occurred during the MSA phase (63%), 
while the fewest responses were received for the TD phase (14%). Overall, the 
adjustments recommended ranged from 11% to 24%. The most adjustments 
occurred during the TD phase (24%), while the E&MD phase received the fewest 
adjustments (11%). 

The second results section pertained to the HSI activity database columns 
that are required to establish a HSI/cost estimation database. The SMEs 
considered all columns were useful for establishing a historical database in 
support of HSI cost estimation activities. 

The third section of results pertained to review of cost drivers. These 
results provided an identification of drivers (type) as well as 33 additional drivers 
that can attribute to an HSI cost estimation process. 

Finally, the SMEs provided HSI-Effort percentages of the life cycle phases 
for a general ACAT I program. These results are intended to provide a starting 
point or baseline for HSI costs. The HSI-Effort (out of 100%) results included the 
following percentage ranges: MSA (3-28%), TD (3-20%), E&MD (2-8%), P&D (4- 
10%), and O&S (5-15%). The remainder of this thesis discusses these results in 
more detail as well as provide recommendations for future work. 
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V. DISCUSSION 


The purpose of this thesis was to develop a qualitative method for 
determining HSI-specific costs that can be used for conducting tradeoffs during 
the AoA. The effort was necessary to improve the accuracy of cost estimates 
over the life cycle of the system and to provide a basis for a new HSI cost 
estimation process. The method produced an HSI activity database, HSI cost 
drivers and an HSI level of effort for every acquisition life cycle phase for a 
generic ACAT I program (See Appendix C). SMEs provided the requisite 
knowledge for refining the database. The following sections discuss the 
qualitative findings. 

A. RESPONDENTS 

Although the sample size was small, the range of SME competency areas 
provided a diverse view of the HSI cost processes. HSI is a relatively young 
discipline with few SMEs. The small pool of SMEs (who tend to be in high 
demand and are typically overworked) may have contributed to the somewhat 
low response rate of approximately 48%. Other factors that contributed to the 
response rate included availability, unfamiliarity with phase assignment, and 
confusion concerning what was expected of the SME. 

Response rates varied within and between services for several reasons. 
For example, USAF SMEs were reassigned during the review and were unable 
to complete their assessment of the database. In these cases, respondents were 
encouraged to provide as many recommendations as time permitted. Therefore, 
analyses were based on individual data item entries instead of individual 
responses. 

Every attempt was made to match a SME’s expertise to the appropriate 
acquisition phase. However, some SMEs reported they were unfamiliar with the 
HSI activities within the acquisition phases to which they were assigned. In these 
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cases, when SMEs appealed to the review administrator, they were offered an 
alternate phase to review. Respondents also gave domain viewpoints (i.e., 
training, manpower) as well as holistic viewpoints. 

B. HSI ACTIVITY DEVELOPMENT 

The HSI database responses provided validation of the activities needed 
to perform cost estimations. During the HSI activity evaluation, SMEs 
emphasized balancing the results of this study of the activities found in the 
system engineering process and human view frameworks. Additionally, SMEs 
suggested that future efforts should clarify HSI activities in the Acquisition 
Management, Acquisition Engineering and Support/Sustainability activity tiers. 
This clarification would necessitate determining “areas of responsibility” within 
the HSI domain practitioner community. SMEs also stated that HSI work was 
usually taken “out of hide” and that HSI work is often dismissed or not 
considered. For example, manpower authorizations determined by 
knowledgeable practitioners have not been properly considered during trade-off 
deliberations. Other SME suggestions included adding items to the database 
such as “Establishing a need for HSI involvement in an IPT” and “HSI planning 
and management” elements. 

Another finding pertained to the specificity of some database items. SMEs 
suggested that some activities contained an excessive amount of details. This 
may have been true for some activities. However, some HSI processes need to 
be described in detail so that the nature of the activity is truly understood. For 
example, modeling & simulation activities or specialized HSI tools may be 
needed in some types of analyses. If these activities and tools are not specified, 
the associated cost elements will not be accurate; they may be underestimated 
significantly. 

The SMEs also suggested including activities such as participation in 
technical information meetings (in the E&MD phase) and contributing to 
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contractor taskings. Contractor tasking is a controversial and obscure cost issue. 
Since contractors perform many of the HSI activities, contracts should include 
indirect costs for these personnel as stated in DTM-09-007 (2010). 

Overall, the data collected from the SMEs supported the use of the 
proposed HSI database for AoA and Test & Evaluation cost estimation 
processes. Although the TD Phase returned the fewest responses, it was the 
most informative for AoA guidance. The SMEs elaborated on activities regarding 
AoA trade-off studies and suggested assessment activities that were integrated 
across HSI domains. These activities demonstrate the need for a significant level 
of HSI practitioner involvement. Test & Evaluation activities were also expanded 
to include demonstration, validation and verification processes. 

C. DATABASE COLUMNS 

The database column review resulted in many positive comments from the 
SMEs. They agreed that all proposed data elements were informative for HSI 
cost purposes. However, they recommended that the “Event/Category” column 
be refined by using a cost tier structure to organize the activities. One SME 
suggested that program IPTs tend to focus on WBS level II and III tiers but that 
cost estimation becomes increasingly difficult at these levels because of the 
detail inherent in these tiers. It may not be feasible to perform cost estimation for 
HSI activities at the WBS tier III during the AoA. It may be that the tier III level of 
analysis will have to wait until later in the acquisition process. 

D. COST DRIVERS 

The cost driver evaluation included a general list of drivers identified 
during the literature review. These drivers consisted of typical HSI domain, cost 
method and System Engineering driver elements. The SMEs reported that the 
cost drivers were appropriate for an HSI cost database. Although, the cost driver 
list was intended for HSI activities, SMEs suggested that several of the driver 

elements could be used in activities unrelated to HSI (e.g., configuration 
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management, SLOC). SMEs also classified the cost drivers as either direct or in 
direct and assigned cost drivers to each activity. 

Having the SMEs assign cost drivers to various activities seemed to be a 
viable approach to cost estimation. However, the SMEs indicated that costs 
rarely fit into such neat packages. In spite of this, the SMEs were able to validate 
the potential drivers for specific activities. 

E. HSI EFFORT 

The HSI Effort data provided the final (and most interesting) findings for 
the AoA cost estimation. SMEs provided HSI effort for overall phase to which 
they were assigned, either as a percentage of total effort or as the number of 
man-hours required. SMEs indicated that the most HSI activity should occur 
during the MSA phase. SMEs also suggested that the HSI participation as a 
whole (within the MSA phase) attributed to approximately 40-60% of effort (out of 
100%) of a general ACAT I program. Within the MSA phase, HSI Effort for 
individual activities ranged from 1% to 28%. Those SMEs who expressed HSI 
participation in terms of man-hours provided some interesting insights. Examples 
of these man-hour estimates included: 

• Provide HSI Assessment based on best available solution 
description (in CBA, AoA, Study Guidance, ICD) across all HSI 
domains to define level of risk in each: 1000 hours. 

• Participate in AoA Assessment: 500 hours. 

• Perform an HSI analysis of legacy systems with respect to each 
domain and address the potential integrated impact and associated 
risks: 500 hours. 

Other findings suggested that ACAT I programs were multiyear efforts, and 
SMEs found it difficult to assess the individual activities across multiple years and 
multiple increments. 
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VI. RECOMMENDATIONS FOR FUTURE WORK AND 

CONCLUSION 


A. RECOMMENDATIONS 

The thesis explored HSI cost activity development for the AoA process of 
a generic ACAT I program. It also provides an initial database that contains the 
HSI activities that should be included in the cost estimation process. The thesis 
focused on cost estimation in support of the AoA but considered HSI activities 
that span the entire acquisition life cycle. Based on the results and analysis 
reported herein, the following recommendations are made for future work. 

Identify the most important HSI activities in each of the oversight 
documents, including the following: Review Acquisition Strategy; Acquisition 
Plan; Source Selection Plan; RFP; Final Product Baseline; Critical Safety List 
Items; PESHE; and TEMP. 

• Perform HSI cost method comparisons in order to develop 
variables for a cost model; 

• Clarify HSI activities in the Acquisition Management, Acquisition 
Engineering and Support/Sustainability activity tiers.; 

• Organize the database in terms of work breakdown structure cost 
elements; 

• Identify the benefits/impacts and risk/cost relationships of HSI 
activities; 

• Determine domain tradeoffs that must occur to achieve accurate 
cost estimates or the AoA; and, 

• Use a system of interest to in a case study approach to verify the 
database. 
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Unfortunately, most programs do not contain HSI cost data. Therefore, a 
majority of HSI cost estimates will have to rely, at least initially, on “rule of thumb” 
or heuristic approaches. Two programs that are making a concerted effort to 
document HSI costs are the MQ-1 and LCS. These programs should provide HSI 
cost data that can be used in a parametric cost estimation approach. The 
parametric approach to estimating HSI costs is the ultimate goal. 

B. CONCLUSION 

Cost estimation is an amalgamation of systems that helps ensure the 
validity, reproducibility and affordability of any defense acquisition program. 
Capability assessments allow comparisons with benchmark resources (current or 
legacy system capabilities and costs). The resultant cost data can then be 
applied to the estimation of costs for future systems as early in the acquisition 
process as the AoA. This thesis was intended to be general in nature so that it 
can be applied to any military service and to any type of system. The research 
effort emphasized issues such as development/operation test and evaluation 
processes, program increments, operation and support, and environmental costs. 
This method will allow HSI practitioners and other professionals the ability to 
assess the level of HSI effort required for an acquisition program and to mitigate 
potential risks and uncertainties that may adversely impact the acquisition of 
systems. The goal is to provide the best possible system to the Warfighter that is 
also cost effective and timely. This thesis is an important step in that direction. 
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APPENDIX A. COMPOSITE STANDARD PAY CHARTS AND 

REIMBURSEMENT RATES 


^ni_ITAR.Y COMPOSITE STA>T)ARD PAY A>T) REIMBT-rRSEMEN'T RATES 


MILITARY 

PAY 

DEPARTMENT OF THE AIR FORCE 

FOR FISCAL YEAR 2011 ‘ 

ANTST AL DOD 

a\t:r-age composite 

ANTST AL RATE 

BILLABLE TO 

OTHER 

FEDERAL 

GRADE 

BASIC PAY 

RATE - 

AGENCIES 

0-10 

$181,587 

$300,725 

$305,184 

0-9 

181,587 

305,777 

310,236 

0-8 

160,654 

273,032 

277,491 

0-7 

139,237 

246.184 

250,643 

O-fi 

115,447 

214.027 

218,486 

0-5 

93,313 

182.049 

186,508 

0-4 

78.866 

160.639 

165,098 

0-3 

63,403 

136,217 

140,676 

0-2 

48,814 

110,344 

114,803 

0-1 

34,673 

87,651 

92,110 

WO-5 

_ 

_ 

_ 

WO-4 

— 

— 

— 

WO-3 

— 

— 

— 

WO-2 

— 

— 

— 

WO-1 

— 

— 

— 

E-9 

$72,291 

$137,410 

$141,869 

E-8 

58,494 

116,749 

121,208 

E-7 

49,755 

104.179 

108,638 

E-6 

40.444 

90.069 

94,528 

E-5 

32,353 

76,538 

80,997 

E-4 

25,667 

63,428 

67,887 

E-3 

21,021 

50,452 

54,911 

E-2 

19,727 

46.434 

50,893 

E-1 

16.900 

41.007 

45,466 

CADETS 

$11,654 

$16,793 

Not applicable 


■Jotes; 

/ Effective fiscal year 2005. military personnel services for Foreign Military Sales (FMS) shall be priced using the Composite Rates 
that already include permanent change of station (PCS) expense and shall no longer use the actual PCS expense for PCS moves to 
support a FMS case. Effective FY 2006. the military personnel services for FMS shall be priced using the DoD Composite Rate 
plus the acceleration factor shown in Tab K-1. Reimbursement of the acceleration factor shall be deposited into the Defense Health 
Progr am (97*0130). Reimbursement of the per capita normal cost for Medicare-eligible retiree health care (MERHC) accrual shall 
be deposited into the Miscellaneous Receipts Account 3041. The next update of die DoD F\1R V'ol. 15 Section 070203 will reflect 
these change. 

t The annua l DoD composite rate includes the follov^mg military personnel appropriation costs; average basic pay plus retired pay 
accrual. Medicare-eligible retiree health care (MERHC) accrual, basic allowance for housing, basic allowance for subsistence, 
incentive and special pay. permanent change of station expenses, and miscellaneous pay. Includes a per capita normal cost of 
$5,673 for MERHC accrual - see Tab K-1 

/ The annual rate billable to Other Federal Agencies recovers additional military related health care coste financed by the Defense 
Health Program. The annual billable rate includes an acceleration factor of $10,132 for all personnel. Excludes per capita normal 
cost of $5,673 for MERHC accrual — see Tab K-l. 

/ To compute a Daily Rate, apply a factor of .00439. To compute an Hourly Rate, apply a factor of .00055. 

/ Basic pay for these officers is limited to the rate of basic pay for Level n of the Executive Schedule, which currendy is $181,587 
per year. 


Tab K-5 


Figure 24. 


(From “Airforce military composite standard pay and 
reimbursement rates,” 2010) 
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^^LIT.4RY COMPOSITE ST.4XDARD PAY .A>T> REIMBfRSEMENT R.A.TES 


^^LIT.4RY 

PAY 

DEP.\RTMENT OF THE 

FOR FISCAL YZ.\R 2011 * 

.AVNF.4L DOD 

.\\t;r.4ge composite 

-ANNL R.ATE 

BILLABLE TO 

OTHER 

FEDER-\L 

GFLADE 

B-A.SIC PAY 

R-4TE - 

-\GENCIES 

0-10 

$181,587 

$291,527 

$295,986 

0-9 

181,587 

293,387 

297,846 

0-8 

161,619 

267,571 

272,030 

0-7 

139.396 

238,269 

242,728 

0-6 

117.298 

218,288 

222,747 

0-5 

94.425 

187.883 

192,342 

0-4 

79.869 

169.549 

174,008 

0-3 

65,118 

144,700 

149,159 

0-2 

51.126 

113,231 

117,690 

0-1 

37,874 

90,901 

95,360 

WO-5 

$95,112 

$182,380 

$186,839 

WO-4 

82,822 

157,931 

162,390 

WO-3 

70.047 

140.858 

145,317 

WO-2 

57,958 

123,935 

128,394 

WO-1 

— 

— 

— 

E-9 

$71,210 

$140,175 

$144,634 

E-8 

56,707 

119,669 

124.128 

E-7 

47,527 

105.918 

110,377 

E-6 

39,346 

92,856 

97,315 

E-5 

31.585 

79.826 

84,285 

E-4 

25,354 

67,296 

71,755 

E-3 

21.516 

55,557 

60,016 

E-2 

19,676 

50,071 

54,530 

E-1 

17,122 

44.599 

49,058 

CADETS 

$11,648 

$17,241 

Not applicable 


Notes: 

1/ Effective fiscal year 2005. military personnel services for Foreign \Iilitary Sales (FMS) shall be priced using the Composite Rates 
that already incltide permanent change of station (PCS) expense and shall no longer use the actual PCS expense for PCS moves to 
si^jport a FMS case. Effective FY 2006. the military personnel services for FMS shall be priced using the DoD Composite Rate 
plus die acceleration factor shown in Tab K-1. Reimbursement of the acceleration factor shall be deposited into the Defense Health 
Program (97*0130). Reimbursement of the per capita normal cost for \Iedicare-eligible retiree health care (\1ERHC) accrual shall 
be deposited into the Miscellaneous Receipts Account 3041. The next update of the DoD FMR Vol. 15 Section 070203 will reflect 
these change- 

2/ The annual DoD composite rate includes the following mihtary personnel appropriation costs: average basic pay phis retired pay 
accrual. Medicare-eligible retiree health care (MERHC) accrual, basic allowance for housing, basic allowance for subsistence, 
incentive and special pay. permanent change of station expenses, and miscellaneous pay. Includes a per capita normal cost of 
$5,673 for MERHC accrual - see Tab K-1. 

3/ The annual rate billable to Other Federal Agencies recovers additional military related health care costs financed by die Defense 
Health Program. The annual billable rate includes an acceleration factor of $10,132 for all personnel. Excludes per capita normal 
cost of $5,673 for MERHC accrual — see Tab K-1. 

4/ To compute a Daily Rate, apply a factor of .00439. To compute an Hourly Rate, apply a factor of .00055. 

5/ Basic pay for these officers is limited to the rate of basic pay for Level n of the Executive Schedule, which currently is $181,587 
per year. 


Tab K-3 

Figure 25. (From “Navy military composite standard pay and reimbursement 

rates,” 2010) 
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APPENDIX B. HEALTH HAZARD COST 


Related cost 
component 

Equation 

variable 

Variable 

value 

Description 

All (except 
death costs) 

Po 

See Table 1 

Hazard Probability (HP) - Probability of 
exposure per year, based on the determined 
hazard probability category 

All (except 
death costs) 

s. 

See Table 1 

Hazard Severity (HS) factor based on the 
determined hazard severity category 

All (except 
death costs) 

Ns 

No. of systems 

Number of systems, the total number individual 
items of materiel, equipment, or weapon 
systems in Army inventory 

All (except 
death costs) 

Np. 

No. of persons 

Number of persons per system, or crew size 
for system, or item 

All (except 
death costs) 

Np 

Calculated 

Total number of people exposed to hazard per 
year for the systems or items 

Clinic costs 

Cp 

Calculated 

Cost of clinic visits 

Clinic costs 

N, 

Calculated 

Number of people injured or ill 

Clinic costs 

N. 

Calculated 

Number of clinic visits 

Clinic costs 

\ 

0.75 

Visit constant as result of exposure. The visit 
constant (Ve) equals 0.75 and is based on expo¬ 
sure to a health hazard that results in illness or 
injury. We assumed that if an exposure event 
occurs, then 75 percent of all persons exposed 
to the hazard will visit the clinic for an examina¬ 
tion to determine whether any injury has 
occurred. 

Clinic costs 


See Table 5 

Incidence of injury or illness based on the 
determined risk level for the individual item of 
materiel 

Clinic costs 

Np 

See Table 6 

Number of visits by injured or ill personnel based 
on the determined hazard severity category. The 
hazard severity category determines the serious¬ 
ness of the medical outcomes that could occur. 
As the severity increases, the number of clinic 
visits increases. For this cost component, based 
on values selected by a panel of experts, we 
assigned the number of visits based on the 
hazard severity category and the potential 
medical outcomes. 

Clinic costs 

Pc 

$122 per visit 

Average fee per clinic visit, based on the 
average of various types of clinic service visit 
fees. We found the average fee was SI 22 per 
clinic visit. 


Figure 26. Equation variables (From "Estimating the health hazard costs,” 

1998) 
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Hospitalization 

costs 

c. 

Calculated 

Cost of hospitalization 

Hospitalization 

costs 

Np. 

Calculated 

Number of persons hospitalized 

Hospitalization 

costs 

N. 

Calculated 

Number of hospital days 

Hospitalization 

costs 

1. 

See Table 7 

Incidence of hospitalization based on the 
determined risk level for the individual item of 
materiel 

Hospitalization 

costs 


See Table 8 

Factor for the average number of days in 
hospital per person based on historical hospital 
stay distribution 

Hospitalization 

costs 


See Table 9 

Factor for the hospitalization population 
distribution for average number of days in 
hospital 

Hospitalization 

costs 

F. 

S1,669 per day 

Average fee per hospital day. Average cost based 
on various types of hospital diagnosis-related 
groups and the classification of the disease. We 
found the average hospital fee was $1,669 per 
day. 

Lost time costs 

c, 

Calculated 

Cost of days of lost time 

Lost time costs 

Np. 

Calculated 

Number of persons losing time 

Lost time costs 

N, 

Calculated 

Number of lost workdays 

Lost time costs 


See Table 10 

Incidence of lost time based on the determined 
risk level for the individual materiel item 

Lost time costs 


See Table 11 

Factor for the number of lost workdays per 
person based on historical lost workday 
distribution 

Lost time costs 

Dk 

See Table 12 

Lost time population distribution based on 
average lost workday distribution 

Lost time costs 

w. 

$53.97 per day 

Average wage per day. We based the average 
wage per day (Wd) on the salaries and numbers 
of persons drawing that salary for a selected 
group of personnel. We determined an average 
wage to be $53.97 per day. 

Lost time costs 

B, 

1.41 

Wage fringe benefit factor. We assigned the fringe 
benefit factor (Bf)a value of 1.41. It is a standard 
factor within the government used for program¬ 
ming personnel budget requirements and is 
representative of other corporate benefit factors. 

Disability costs 

c. 

Calculated 

Cost of disabilities 


Figure 27. Equation variables continued (From “Estimating the health hazard 

costs,” 1998) 
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Disability costs 

p<J 

Calculated 

Number of persons disabled 

Disability costs 

1, 

See Table 13 

Incidence of VA disability based on the deter¬ 
mined risk level for the individual item of 
materiel, equipment, or weapon system 

Disability costs 

T, 

0.25 

VA disability adjustment factor for delayed 
disability (5 years/20 years) 

Disability costs 

D. 

See Table 14 

VA disability population factor based on 
historical rate of disability distribution 

Disability costs 

B, 

See Table 15 

VA disability compensation factor per month 
per rate of disability 

Disability costs 

I. 

0.001 

Incidence of active-duty temporary disability 
(1 case/1000 persons) 

Disability costs 

B, 

$9,242 per 
person 

Active-duty temporary disability compensation 
per year 

Disability costs 

'p 

0.011 

Incidence of active-duty permanent disability 
(11 cases/1000 persons) 

Disability costs 

Bp 

$12,864 per 
person 

Active duty permanent disability compensation 
per year 

Rehabilitation 

costs 

c, 

Calculated 

Cost of rehabilitation 

Rehabilitation 

costs 

Nr 

Calculated 

Number of rehabilitation cases 

Rehabilitation 

costs 

D 

r 

See Table 16 

Eligible VA disability population factor based on 
rate of disability distribution equal to or greater 
than 20 percent 

Rehabilitation 

costs 

Qr 

0.05 

VA rehabilitation qualification factor 
(5 cases/100 persons eligible) 

Rehabilitation 

costs 

Br 

$12,000 per 
year per person 

VA rehabilitation benefit per year per person. We 
estimated to be $12,000 per year per person. 
Rehabilitation benefits may vary per person, but 
we considered $12,000 to be a reasonable 
estimate. Other benefits may be available for 
eligible disabled persons, but we did not consider 
these other benefits. 

Death costs 


Calculated 

Cost of death 

Death costs 

N<. 

See Table 17 

Number of deaths per year 

Death costs 

B<^ 

$200,000 

Death benefit and expenses 


Figure 28. Equation variables continued (From “Estimating the health hazard 

costs,” 1998) 
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Table 5. Incidence of Illness or Injury (I,) for System Risk Categories 


System risk category 

Incidence rate (I,) 

High 

0.122 

Medium 

0.095 

Low 

0.067 


Table 6. Number of Clinic Visits (N^) for Hazard Severity Categories 


Hazard severity category 

Number of clinic visits (NJ 

1 

5 

II 

3 

III 

2 

IV 

1 

Table 7. Incidence of Hospitalization (l,^) for System Risk Categories 

System risk category 

Hospitalization rate (l„) 

High 

0.013 

Medium 

0.007 

Low 

0.0005 

Table 8. 

Factors for Average Number of Days in Hospital (D^^) (days/person) 

Length of stay in hospital 

Factor 

<2 days 

1.0 

2-5 days 

3.5 

6-30 days 

18.0 

>30 days 

30.0 


Table 9. Factors for Hospitalization Population Distribution (D^^) 
by Length of Stay in Hospital for System Risk Categories 


System risk category 

Length of stay in hospital 

<2 days 

2-5 days 

6—30 days 

>30 days 

High 

0.40 

0.35 

0.17 

0.08 

Medium 

0.40 

0.36 

0.18 

0.06 

Low 

0.42 

0.37 

0.20 

0.02 


Figure 29. Health Hazard Cost Method Criteria (From “Estimating the health 

hazard costs,” 1998) 
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Table 10. Incidence of Lost Time (I,) for System Risk Categories 


System risk category 

Lost time rate (1,) 

High 

0.055 

Medium 

0.054 

Low 

0.028 


Table 11. 

Factors for Average Number of Days of Lost Time (D,^) (days/persen) 


Number days of lost time 

Factor (D,^) 

<2 days 

1.0 

2-5 days 

3.5 

6-30 days 

18.0 

>30 days 

30.0 


Table 12. Factors for Lost Time Population Distribution (D„) 
by Days of Lost Time for System Risk Categories 


System risk category 

Lost time 

<2 days 

2-5 days 

6-30 days 

>30 days 

High 

0.22 

0.30 

0.29 

0.20 

Medium 

0.20 

0.33 

0.31 

0.16 

Low 

0.15 

0.43 

0.38 

0.04 


Table 13. Incidence of VA Disability (1^) for System Risk Categories 


System risk category 

VA disability factor (IJ 

High 

0.032 

Medium 

0.012 

Low 

0.00005 


Table 14. Factors for Disability Population Distribution {D^) 
by Degree of Disability for System Risk Categories 


System risk category 

Degree of disability 

10% 

20%-50% 

60%-90% 

100% 

High 

0.44 

0.42 

0.10 

0.04 

Medium 

0.44 

0.44 

0.09 

0.03 

Low 

0.43 

0.48 

0.08 

0.01 


Figure 30. Health Hazard Cost Method Criteria (From “Estimating the health 

hazard cost,” 1998) 
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Table IS. VA Disability -Cempeiifatian iFnct«rs |B | by Degree 
el Disabilily {deller i/nienth/per&en) 


Degree of disability 

VA disability comf>ensa1iDn factor J 

10*}^ 

satjoa 


sascias 


set EL50 

tOCRt 

11.865.00 


Table 16. Eligible VA Disability Pepulatien Distributien Factors |D,) 
by Degree «f Disability for System Risk Categoriei 


Syslem risk cate^ry 

10% 

Degree nf disability 
20%-90% eo%^% 

100% 

High 

OJO 

0.42: 

0.10 

0.04 

Mecium 

0.0 

0.44 

0.00 

0.03 

Lew 

OJO 

oja 

0.08 

0.01 


Table IT. Number of Deaths (N^ for Hazard Severity Categeriei 


Hazsnl severity calegcty 

Nunber of deaths (H^ 

1 

1 

II 

0 

III 

0 

IV 

0 


Table 16. Health Hazards and Associated Risk Indices for System X 


l-liKZdn] 

Risk 

asatwmcfYt 
Riujvd code (RACI 

hLazand 

■sevErity 

category 

hlazard 

p«o4s<abSty 

ChiefniciaJ sutstanoea. 

Wcapona co4nbii9.1io4i products 

1 

1 

A 

Chrcmicdi 34jh4lanc)» 

Fr? ntingui^iihg Dgenta 

2 

II 

C 

Cl’icfnicdl substaiMDca 

Carbon {StKidc- 

3 

II 

□ 

AcouaicaJ 

ImpulK [MHK- 

2 

II 

c 

Ac«j:-sticai 

Stcady-aUitE noHse 

2 

II 

c 

T«npt™tijre oclremea 

Cold StFE-H. 

2 

■ 

c 

Tempernture oclremea 

HeoT alxE-H. 

2 

II 

c 

Oxygen ddicieocy 

Oxygen defic*ency (veoblatioo) 

2 

II 

c 

FladiBtio4i crrcrgr^ 

HonionLxing raciat»n 

2 

II 

c 

Fladistiofi energy 

iDnizirig rat&adoci 

4 

II 

E 


Figure 31. Health Hazard Cost Method Criteria (From “Estimating the health 

hazard cost,” 1998) 
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APPENDIX C. LIFE CYCLE PHASE HSI ACTIVITIES 


Event 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

A. 

Provide FISI activity input 
during Capabiiities Based 
Assessment for ICD 

*DOTMLPF Analysis, 
Analysis of Materiel and 
Non-Materiel, and lessons 

Pre-Materiel 

Solution 

Analysis 

All Domains 

(Integrated 

Assessments) 

Documentation to match life 
cycle needs (Breadth and 
depth of required 

B. 

Provide FISI input into 

HSIP for the System 
Engineering Pian (SEP) 

*Appoint HSI Lead. 

**Work with PM to 
determine if standalone 

Pre-Materiel 

Solution 

Analysis 

All Domains 

(Integrated 

Assessments) 

Compiexity, Maintenance 
Concept, Manning, Mission, 
Operational Capability, 

C. 

Provide a total system 
analysis based on 
functionai reiationship 

*Analyze and assess trade 
space, related 
constraints/requirements 

Pre-Materiel 

Solution 

Analysis 

All Domains 

(Integrated 

Assessments) 

Complexity, Maintenance 
Concept, Manning, Mission, 
Operational Capability, 

D 

Provide HSI considerations 
into AoA Study pian 

*ldentify AoA manpower 
estimating tools, analysis 
techniques. 

Pre-Materiel 

Solution 

Analysis 

All Domains 

(Integrated 

Assessments) 

General Drivers: 

Complexity, Maintenance 
Concept, Manning, Mission, 

E. 

Provide input into ESOH 
integration Strategy. 

*Define HSI roles and 
activities for Environmental, 
Safety and Occupational 

Pre-Materiel 

Solution 

Analysis 

All Domains 

(Integrated 

Assessments) 

Complexity, Maintenance 
Concept, Manning, Mission, 
Operational Capability, 

F. 

Provide a HSI assessment 
based on best available 
solution description (in 

*lnfrom Sustainment and 
affect design the design 
and support plans that 

Pre-Materiel 

Solution 

Analysis 

All Domains 

(Integrated 

Assessments) 

Manning 

E. 

Perform a HSI analysis of 
legacy systems with respect 
to each domain and 

*Review the legacy 
system's Post 
Implementation Review 

Materiel 

Solution 

Analysis 

All Domains 

(Integrated 

Assessments) 

Cognitive Workload, 
Complexity, Initial Training 
(Instructor & Key 

F. 

Contribute to assessment 
and development of system 
concepts /Materiel 

^Consider personnel 
alternatives based on AoA. 
**Determine appropriate 

Materiel 

Solution 

Analysis 

All Domains 

(Integrated 

Assessments) 

Initial Training (Instructors 
Key Personnel), Initial 
Training (New Equipment at 

G. 

Assist in development of 
cost estimate based on 
anticipated HFE, 

*Provide input to Manpower 
Estimate Reports 

Materiel 

Solution 

Analysis 

All Domains 

(Integrated 

Assessments) 

Initial Training (Instructors 
Key Personnel), 
Maintenance Requirement, 

H. 

Conduct Mission Task 
Analysis. 

*Define the human 
performance characteristics 
of the user population. 

Materiel 

Solution 

Analysis 

All Domains 

(Integrated 

Assessments) 

Cognitive Workload, 
Complexity, Criticality of 
Task, Difficulty of Task, 


Figure 32. Materiel Solution Analysis HSI Activities 


127 



















Event 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

I. 

Conduct Initial Top Down 
Requirements Analysis. 

*Participate in an IPT and 
provide preliminary input 
into Manpower and 

Materiel 

Solution 

Analysis 

Manpower 

Cognitive Workload, 
Frequency of Task, 

Manning, Mission, 

J. 

Assist in performance of 
Manpower /Personnel 
alternatives and concept 

*Use findings from mission 
task analysis and other 
personnel assessments. 

Materiel 

Solution 

Analysis 

Personnel, 

Manpower 

Operational Capability, 
Personnel 

experience/continuity 

K. 

Identify manning needs, 
manpower lessons learned, 
and high drivers from 

*Establish preliminary 
manpower thresholds and 
objectives. 

Materiel 

Solution 

Analysis 

Manpower, 

Personnel 

Complexity, Maintenance 
Requirement, Manning, 
Mission, Operational 

L. 

Assist in linear modeling 
and studies analysis of 
system approach 

*Assist in Effectiveness 
Analysis Working Group 
(EAWG). 

Materiel 

Solution 

Analysis 

Manpower, HFE, 
Training 

Manning, 

Modeling&Simulation, 
Operational Capability, 

M. 

Conduct 

Manpower/T raining 
assessments to determine 

*Conduct preliminary 
learning performance and 
solution analysis to justify 

Materiel 

Solution 

Analysis 

Training, 

Personnel 

Cognitive Workload, Initial 
Training (Instructor & Key 
Personnel), Initial Training 

N. 

Assist in deriving 

Systematic Prediction of 
Conditions through Initial 

*Review draft /final 
assessments and 
associated costs. 

Materiel 

Solution 

Analysis 

ESOH 

Criticality of Task, 

Frequency of Task, 

Manning, Mission, Physical 

O. 

Participate in AoA 
assessments. 

*Optimize MPT for AoA. 
**ldentify capability gaps in 
Manpower reduction. 

Materiel 

Solution 

Analysis 

All Domains 

(Integrated 

Assessments) 

Cognitive Workload, 
Complexity, Maintenance 
Requirement, Manning, 

P. 

Define concept capabilities 
and requirements. 

*Determine HFE and 

Human performance 
requirements. 

Materiel 

Solution 

Analysis 

HFE, Manpower, 
Personnel 

Cognitive Workload, 
Complexity, Frequency of 
Task, Maintenance 

Q. 

Provide HSI input into 
Technology Development 
Strategy. 

*Provide preliminary inputs 
into MDA cost estimates for 
TD phase to support the 

Materiel 

Solution 

Analysis 

HFE, Manpower, 

Personnel, 

Training 

Cognitive Workload, 
Complexity, Criticality of 
Task, Difficulty of Task, 

R. 

Provide HSI Input into Life- 
cycle Sustainment Plan. 

*Provide preliminary 
considerations into 
Habitability, Training, and 

Materiel 

Solution 

Analysis 

All Domains 

(Integrated 

Assessments) 

Demilitarization and 
Disposal, Migration 
complexity (Influence of 

S. 

Provide HSI input into 
Support System 

Engineering Technical 

*Provide input into the Initial 
Technical Review (ITR) and 
Alternative System Review 

Materiel 

Solution 

Analysis 

HFE, Manpower, 
Training 

Complexity, Manning, 
Mission, Requirements 
understanding (Subjective 


Figure 33. Materiel Solution Analysis HSI Activities (continued) 
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EVENT 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

GUIDE 

Review all the inputs to 
the Technology 
Development Phase 

1.0 Update HSI 
domain effects. 

1.1 Review and update 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning, tools, 
simulation, software, 
analyses, training. 

A. 

Determine the need for 
an HSI IPT or a 

Working Group. 

*Review outputs from 
user CONOPs 
developments, needs 

Technology 

Development 

All as 

appropriate to 
the solution 

Manning, tools, 
simulation, software, 
analyses, training. 

B. 

Establish a HSI IPT or 
Working Group - 
consistent with the size 

*ldentify candidate 
participants / team 
members to participate 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning, Personnel 

experience/continuity 

(Subjective 

GUIDE 

Throughout the next 5 
steps C-G conduct 
trade offs and support 

* Participate in AoA to 
ensure that HSI 
considerations have 

Technology 

Development 

All Domains 

(integrated 

assessment) 


C. 

Determine all relevant 
critical operational 
issues and determine 

* Interpret User Needs, 
Analyze Operational 
Capabilities & 

Technology 

Development 

HFE, 

Manpower, 

Training, 

Maintenance Concept, 
Maintenance 
Requirement, Mission, 

D. 

Callout 

Develop system 

performance 

(&constraints) 

* Ensure HSI criteria 
are traceable back to 
defined system 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning, travel. 

E. 

Callout 

Develop functional 
definitions for 
Enabling/Critical 

* Define HSI criteria for 
weapon system, 
support, equipment. 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning, tools, 
evaluations, analyses. 

F. 

Callout 

Decompose functional 
definitions into Critical 
Component Definition 

* Update system HSI 
criteria. 

* Assess HSI impacts 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning, simulation, 
tools, evaluations. 

G. 

Identify HSI technical 
risks and mitigation 
strategies for 

*ldentify detailed 
administrative 
processes for HSI 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Difficulty of Task, 
Maintenance 
Requirement, Physical 


Figure 34. Technology Development Phase HSI Activities 
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EVENT 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

GUIDE 

Throughout the next 7 
Steps H-O conduct 
trades that are spread 

* Conduct trade 
studies on threshold 
and objective levels of 

Technology 

Development 

All Domains 

(integrated 

assessment) 


H. 

Demonstrate 

enabling/critical 

technology 

* Integrate evaluations 
of critical technologies 
across all functional 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Maintenance Concept, 
Manning, Mission, 
Operational Capability, 

I. 

Provide integrated HSI 
assessment 
methodology for 
technology down 

*Define relevant 
criteria across the HSI 
domains and for 
overall integration 

Technology 

Development 

Human Factors 

Engineering 

(HFE) 

Complexity, Manning, 
Modeling&Simulation, 
Testing Type- 
campaign or force- 

J. 

Refine and update 
requirements. 

*Update HFE for HMI, 
HCI, Workspace, 
Human Performance 

Technology 

Development 

HFE, 

Manpower, 

Personnel, 

Manning 

K. 

Assist in technology 
selection. 

*Address all relevant 
requirements that have 
HSI implications. 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning, Specialty 
Personnel, Training 
Analysis and Design 

L. 

Perform analysis and 
recommendations for 
system technology 

*Perform Operational 
Workflow Analysis. 
*Perform Manpower, 

Technology 

Development 

HFE, 

Manpower, 

Personnel, 

Cognitive Workload, 
Criticality of Task, 
Difficulty of Task, 

M. 

Assist in development 
of System Functional 
Specs & Verification 

*Perform HSI 

Functional Allocation 
Analysis. 

Technology 

Development 

HFE, 

Manpower, 

Personnel, 

Cognitive Workload, 
Manning, Mission, 
Operational Capability, 

N. 

Participation in 
Integrated Test Team 
(ITT) and evaluation or 

*Determine scope of 
Safety and 

Occupational Health 

Technology 

Development 

Environment, 

HFE, 

Manpower, 

Complexity, Manning, 
Modeling&Simulation, 
Testing Type- 

O. 

Develop T&E / V&V 
program plans 
inclusive of all the 

*Develop HSI 

Methods, Metrics, 
Analysis and Tools. 

Technology 

Development 

HFE 

Complexity, Manning, 
Mission, 

Modeling&Simulation, 


Figure 35. Technology Development Phase HSI Activities (continued) 
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EVENT 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 


Guide 

Throughout the next 7 
Steps P-S conduct 
trades that are spread 

* Conduct trade 
studies on threshold 
and objective levels of 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning, modeling / 
simulation, analyses, 
trade studies. 


P. 

Callout 

Demonstrate 

enabling/critical 

technology 

* Integrate evaluations 
of critical technologies 
across all functional 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning, travel, tools. 


Q. 

Callout 

Demonstrate system & 
prototype functionality 
versus plan. 

* Evaluate critical 
technologies from an 
HSI perspective 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning 


R. 

Callout 

Demo/Model 
integrated system 
versus performance 

* Evaluate critical 
technologies from an 
HSI perspective. 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning, models / 
simulation 


S. 

Callout 

Demonstrate & 
validate System & 
Technology Maturity 

* Ensure applicable 

HSI elements are 
embedded in the 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning 


GUIDE 

Insure the production 
and completion of 
inputs to the following 

1.0 Incorporate domain 
considerations into 
baseline parameters 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning, tools, 
simulation, software, 
analyses, training. 


T. 

Review and provide 
input into Preliminary 
design and allocation 

*Provide HSI inputs 
into all relevant areas 
(e.g. systems safety. 

Technology 

Development 

Environment, 

HFE, 

Occupational 

Manning, tools, 
simulation, software, 
analyses, training. 


U. 

Provide HSI input into 
the Capability 
Development 

*Provide design 
considerations for 
environmental facts. 

Technology 

Development 

Environment, 

HFE, 

Manpower, 

Manning, Mission 


V. 

Provide input into the 
Acquisition Strategy for 
goals, objectives and 

*Provide an outline 
methodology to: 
**Determine 

Technology 

Development 

HFE, 

Manpower, 

Personnel, 

Manning, Mission, 
Operational Capability 


Figure 36. Technology Development Phase HSI Activities (continued) 
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EVENT 

HSLActivity 

HSLSubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

W. 

Provide input into Acquisition Program 
Baseline (APBs). 

‘Provide input for KPP, KSA, Threshold and Objective values 
and requirements considerations. 

‘Operationally define performance parameters, measures of 

Technology 

Development 

HFE, 

Manpower, 

Personnel 

Level of service requirements 
(Subjective difficulty of satisfying the key 
performance Parameters), Manning, 

X. 

Incorporate requirement for contractor 
generated HSIP(s) and associated 
source selection criteria in Request for 
Proposals (RFPs). 

‘Review RFP language, CDRLs and DIDs for HSI Planning 
activities for each human performance. 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning 

Y. 

Conduct formal development of 
human-centered source selection 
criteria for RFPs at Milestone (MS) A. 

‘Conduct HSI review of source selection criteria for 
developmental RFPs. 

“Review all source selection criteria for HSI relevance. 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Complexity, Maintenance Requirement, 
Manning, Mission, Platforms, Specialty 
Personnel 

Z. 

Apply analysis in findings to Test and 
Evaluation Strategy (TES), LCMP and 
TEMP 

‘Conduct Special analysis to update thresholds, objectives, 
and evolving criteria for DT&E and eventually OT&E. 

‘Update HSI test plans as necessary 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning 

AA. 

Provide input into System Engineering 
Review Process (SRR). 

‘Provide input into SRR for implication of humans into system. 
SFR for operation level functional requirements as IPT 
member. 

Technology 

Development 

HFE, 

Manpower, 

Survivability 

Complexity, Maintenance Requirement, 
Manning, Mission, Operational 

Capability, Personnel/team capability 

BB 

Provide input into System Engineering 
Review Process (SFR) 

‘ Address HSI requirements in the system functional baseline 
and in conjunction with the lower-level performance 
requirements. 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning 

CC 

Provide input into System Engineering 
Review Process (PDR) 

‘ Ensure domain-specific performance requirements are 
included in the preliminary design. 

‘ Review subsystem requirements to address HSI issues 

Technology 

Development 

All Domains 

(integrated 

assessment) 

Manning, Travel 


Figure 37. Technology Development Phase HSI Activities (continued) 
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Event 

HSI_Activity 

HSI_Sub Activity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

A. 

Provide HSI input to 
the TEMP. 

*lncorporate use of 
HSI IPT (Evaluation 
Plans); Identify 

E&MD: Integrated 
System Design 

All Domains 

(Integrated 

Assessment) 

Complexity, 

Manning, 

Mission, 

B. 

Finalize HSI 
requirements by 
obtaining 

*Determine 

Assessment 

methods; 

E&MD: Integrated 
System Design 

All Domains 

(Integrated 

Assessment) 

Complexity, 
Manning, Mission, 
Model ing&Simulati 

C. 

Assist in 

development of 
detailed design 

*Provide guidance 
for supporting 
development of 

E&MD: Integrated 
System Design 

All Domains 

(Integrated 

Assessment) 

Complexity, 
Frequency of Task, 
Mission, 

D. 

Assess detailed 
system designs to 
ensure that they 

^Validate 
manpower 
estimate basis 

E&MD: Integrated 
System Design 

All Domains 

(Integrated 

Assessment) 

Manning, Mission, 

Operational 

Capability, 

E. 

Participate in 

Prototypes/Test 

Materials 

Interactively 
develop and review 
prototypes. 

E&MD: Integrated 
System Design 

Human Factors 
Engineering 

Cognitive 

Workload, 

Complexity, 

F. 

Evaluate Prototypes 
(or Test Materials) 
for HSI criteria . 

Perform heuristic 

evaluations. 

Cognitive 

E&MD: Integrated 
System Design 

All Domains 

(Integrated 

Assessment) 

Cognitive 

Workload, 

Complexity, 

G. 

Participate in design 
demonstrations (to 
include 

*Verify detailed 
system designs. 
*Evaluate initial 

E&MD: Integrated 
System Design 

All Domains 

(Integrated 

Assessment) 

Mission, 

Operational 

Capability, 

H. 

Evaluate HSI criteria 
for the system 
during component / 

*Complete data 
collection during 
DT&E and OT&E for 

E&MD: Integrated 
System Design 

All Domains 

(Integrated 

Assessment) 

Complexity, 
Platforms, Testing 
Type-campaign or 

I. 

Perform HSI 

evaluation on 
system 

*Determine impact 
on human 
performance and 

E&M D:System 
Capability & 
Manufacturing 

All Domains 

(Integrated 

Assessment) 

Complexity, 
Modeling&Simulati 
on. Personnel/team 


Figure 38. Engineering and Manufacturing Development Phase HSI Activities 
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Event 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

J. 

Assist in 

Development of 

Test Plans for 

^Develop test plans 
for human 
performance 

E&MD:System 
Capability & 
Manufacturing 

Human 

Factors 

Engineering 

Occupational 
Training, Testing 
Type-campaign or 

K. 

Participate in HSI 
Test events. 

*Perform analysis 
and reporting 
functions related to 

E&MD:System 
Capability & 
Manufacturing 

All Domains 

(Integrated 

Assessment 

Modeling&Simulati 
on. Operational 
Capability, Testing 

L. 

Participate in ECP 
development. 

^Generate and 

review HFE related 

ECPs for human 

E&MD:System 
Capability & 
Manufacturing 

Human 

Factors 

Engineering 

Manning 

M. 

Provide HSI updates 
to Acquisition 
Strategy and APB. 

^Ensure continued 
emphasis on HSI 
requirements met 

E&MD:System 
Capability & 
Manufacturing 

Manpower 

Personnel, 

Training 

Manning 

N. 

Provide HSI input 
into Capability 
Production 

^Provide input to 
CPD based on 

lessons learned 

E&MD:System 
Capability & 
Manufacturing 

Environme 

nt. 

Habitability 

Manning 

O. 

Provide HSI input 
into Technical 

Reviews and 

^Contribute HSI- 
related entry and 
exit criteria for 

E&MD:System 
Capability & 
Manufacturing 

Human 

Factors 

Engineering 

Manning, 

Operational 

Capability, 

P. 

Provide HSI input 
for Error and Fault 
analysis. 

*Examine workload 
levels at micro-, 
meso-, and macro- 

E&MD:System 
Capability & 
Manufacturing 

Human 

Factors 

Engineering 

Complexity, 
Criticality of Task, 
Difficulty of Task, 


Figure 39. Engineering and Manufacturing Development Phase HSI Activities (continued) 
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Event 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

A. 

HSI Planning and 

Management 

*As member of Integrated 
Product Team, review, update 
and trade-off prior 

Production 

and 

Deployment 

All Domains 
(Integrated) 

Manning, Requirements 
understanding. Level of 

Service Requirements, Time 

B. 

Review and develop HFE 
installation Criteria. 

‘Consider human performance 
workload and safety. 

‘Identify gaps or potential 

Production 

and 

Deployment 

HFE 

Complexity, Level of service 
requirements (Subjective 
difficulty of satisfying the key 

C. 

Verify HSI requirements for 
installation criteria. 

‘Assist in HSI verification of 
system installation 
requirements 

Production 

and 

Deployment 

All Domains 
(Integrated) 

Cognitive Workload, Level of 
service requirements 
(Subjective difficulty of 

D. 

Assist in T&E HSI analyses of 
system design and installation 
(including LRIP deficiencies). 

‘Document HSI-related system 
performance issues (i.e., 

PTRs). 

Production 

and 

Deployment 

All Domains 
(Integrated) 

Cognitive Workload, 

Complexity, Manning, 

Migration complexity (Influence 

E. 

Assist in T&E analyses of 
system design and installation 
(including LRIP deficiencies), 

‘Evaluate system components 
having an impact on human 
performance and safety. 

Production 

and 

Deployment 

Human 

Factors 

Engineering 

Complexity, Level of service 
requirements (Subjective 
difficulty of satisfying the key 

F. 

T&E support and participation 
(lOT&E). 

‘Develop user assessment for 
lOT&E 

“Develop user-centric 

Production 

and 

Deployment 

All Domains 
(Integrated) 

Complexity, Level of service 
requirements (Subjective 
difficulty of satisfying the key 

G. 

T&E support and participation 
(LFT&E) 

(separated from previous 

‘Define and test appropriate 
metrics (identified in JCIDS 
(CDD and CPD), RFP, etc). 

Production 

and 

Deployment 

All Domains 
(Integrated) 

Manning: Testing Type- 
Campaign or force-level. 
Documentation to match Life 

H. 

T&E support and participation 
(FOT&E) 

(separated from previous 

‘Develop user-centric 
assessment to support 

FOT&E. 

Production 

and 

Deployment 

All Domains 
(Integrated) 

Cognitive Workload, 

Complexity, Manning, 

Migration complexity (Influence 


Figure 40. Production and Development Phase HSI Activities 
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Event 

HSLActivity 

HSLSubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

1 . 

Review OPTEVFOR reports and discuss 
HSi trade-offs. 

‘Participate in an IPT and coordinate with 
stakehoider. 

‘Verify and vaiidate finai HFE reiated 
production items. 

Production and 
Deployment 

Human Factors 
Engineering 

Manning, Testing type. Requirements 
Understanding, Documentation to match 
life cycle needs. Multisite coordination 

J. 

Document and anaiyze post depioyment 
iessons iearned. 

‘Coiiect and anaiyze user feedback and 
service use data from post-depioyment 
system. 

“Anaiyses of LRIP system performance 

Production and 
Deployment 

Human Factors 
Engineering 

Manning, Documentation to match life 
cycle needs. Requirements understanding. 
Level of service requirements. Testing, 

Time constraint for System Development 

K. 

Provide input into review process for P&D 
Phase. (OTRR,AOTR,PCA; ‘overview and 
invoived participation) 

‘Provide input for assurance of operationai 
capabiiities and effectiveness (may deiay 
FRP) prior to iOT&E (OTRR)At 
demonstration (AOTR). 

Production and 
Deployment 

Human Factors 
Engineering, 
Survivability, 
Training 

Complexity, Manning, Mission, Operational 
Capability, Personnel/team capability 
(Subjective assessment of the team’s 
intellectual capability). Platforms; Testing 

L. 

Update Manpower/Personnei requirements 
for system design and LCMP. 

‘Perform anaiysis and reporting on 
Manpower reiated test events, data 
coiiected, and findings. 

‘Revise the MER to reflect system design. 

Production and 
Deployment 

Manpower, 

Personnel 

Complexity, Manning, Mission, Platforms, 
Specialty Personnel 

Initial Training (Instructor and Key 
Personnel), Mission, Testing Type- 

M. 

Update Training systems requirements for 
system design and LCMP. 

‘Assess training solutions during initial 
system component demonstrations. 

‘Refine training solutions based on findings 
from initial system component 

Production and 
Deployment 

Manpower, 

Personnel, 

Training 

Initial Training (Instructor & Key 

Personnel), Initial Training (New Equipment 
at system fielding). Manning, Mission, 
Occupational Training, Training Analysis 

N. 

Assist with System Hazard Anaiysis 
(SHA)and Operating and Support Hazard 
Anaiysis (O&SHA) and System Safety 
Anaiysis (SSA). 

‘Participate in system test events. 

“Perform analyses and reporting on Safety 
and Occupational Health (Health Hazards) 
related test events, data collected, and 

Production and 
Deployment 

HFE, Habitability, 
ESOH (Health 
Hazards, Systems 
Safety) 

Manning, Testing Type-system simulation 
and stimulation. Specialty Personnel 


Figure 41. Production and Development Phase HSI Activities (continued) 
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HSI_Activity 

HSLSubActivity 

Acquisition 
Management Phase 

Domain 

Cost Drivers 

A. 

HSI Planning and 
Management 

*Provide updates to the HSI 
Plan. 

*Provide Status reports and 

Operation & 
Support 

All Domains 

(Integrated 

Assessment) 

Manning, Requirements 
understanding. 
Documentation to match life 

B. 

Provide follow-on Test and 
Evaluation support. 

*Collect and analyze user 
feedback and service use 
data from the post-deployed 

Operation & 
Support 

Human Factor 
Engineering 

Cognitive Workload, 
Complexity, Criticality of 
Task, Difficulty of Task, 

C. 

Provide analysis for HFE 
FOC support and 
logistics/sustainment 

*Provide HFE based 
recommendations for 
system modernization and 

Operation & 
Support 

Human Factor 

Engineering 

Habitibility 

Cognitive Workload, 
Complexity, Criticality of 
Task, Difficulty of Task, 

D. 

Provide recommendations 
for HFE FOC support and 
logistics/sustainment 

*Provide HFE based 
recommendations for 
system modernization and 

Operation & 
Support 

All Domains 

(Integrated 

Assessment) 

Complexity, Maintenance 
concept. Maintenance 
Requirement, 

E. 

Provide Training FOC 
support and 
logistics/sustainment 

*Transition post-fielding 
Training Evaluation 

Analysis (PFTEA). 

Operation & 
Support 

Training 

Complexity, Manning, 
Modeling & Simulation, 
Difficulty of Task, Criticality 

F. 

Provide 

Manpower/Personnel FOC 
support and 

*Verify measure of 
effectiveness survey. 

*Verify system installations 

Operation & 
Support 

Manpower, 

Personnel 

Manning (Maintenance 
concept. Requirements 
understanding. Migration 

G. 

Analyze and prototype post¬ 
product improvements for 
the next inremental build. 

*Assess Human-Machine 
interfaces, Human- 
Computer Interfaces, 

Operation & 
Support 

Human Factor 
Engineering 

Cognitive Workload, 
Complexity, Criticality of 
Task, Difficulty of Task, 

H. 

Provide input for 
sustainment of operational 
readiness and safety of 

*Provide modeling and 
simulation process or 
improvement during FRP 

Operation & 
Support 

Human Factor 

Engineering, 

Manpower, 

Manning, 

Model ing&Simulation, 
Testing Type-campaign or 

I. 

Provide testing and 
verification of follow-on T&E 
processes. 

Verify suitability and 
effectiveness for system. 
*(lnclude availability. 

Operation & 
Support 

Human Factor 

Engineering, 

Manpower, 

Manning; Maintenance 
concept. Requirements 
understanding, Physial 

J. 

Provide FOC support and 
logistics/sustainment 
transistion and Operation 

*Provide updates for ESOH 
strategy and PESHE in 
conjuction with SE 

Operation & 
Support 

Environment, 
Occupational 
Health, System 

Manning; Requirements 
understanding. Speciality 
Personnel, Maintenance 

K. 

Provide input into O&S 
review processes (ISR) 

*Provide measurable 
assessments for risk, 
readiness, technical status 

Operation & 
Support 

Human Factor 

Engineering, 

Manpower, 

Manning (Modeling & 
Simulation, Mission, 
Migration Complexity, 


Figure 42. Operation and Support HSI Activities 
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APPENDIX D. REVIEW INSTRUCTIONS 


(Note: For THESIS-VIEW ONLY of Review Instructions. For examples, refer to Figures 42 

and 43, and Tables 27 and 28) 


Human System Integration Activity SME Review 

Background: 

Research is underway at the Naval Postgraduate School to facilitate Human System 
Integration (HSI) cost estimation (CE) in the Analysis of Alternative (AoA) portion of the Defense 
Acquisition process. The goal is to develop a database of HSI CE activities that spans the entire 
Acquisition life cycle and supports the cost estimation processes. 

You have been identified as a Subject Matter Expert (SME) in HSI, an HSI domain or 
cost estimation. We request you review a selected (assigned) portion of the database and provide 
us feedback. Any questions related to this request can be directed to LT Deborah Sindall, USN at 
dasindal@nps.edu and by telephone 571.830.4971. 

Instructions to the Reviewer: (2 Parts) 

1. Column Headings for Access Database. 

Review the columns that are proposed for the Access database. Refer to Excel file Tab 1 
(Column Descriptions) or Table 1 on Page 4 of these instructions for a list of columns and 
descriptions. 

a. On Excel file Tab 1 (Column C), provide comments regarding additions, deletion or 
modification of columns for the database. An example is provided on Page 3 of 
these instructions. Note: Data are not system specific. 

b. If a column should be added, add the item to the bottom of the list. 

2. HSI Activities in Defense Acquisition Phases. 

For selected rows and columns on Excel file (Tab 2, HSI Activity), review the content to 
ensure it is appropriate for cost estimation of HSI activities in AoA. A blank row for each 
activity has been provided for your comments. If no correction is needed, please indicate 
this in the row provided. Examples of comments and corrections have been provided on 
Page 3. Note: You will be given one or two phase activities on the Excel file for 
review. 

a. For Column A (HSI Activity) and B (HSI Sub Activity): 

i. For each row, determine if this is a major HSI activity. If so, is the activity 
worded properly? If incorrect, provide a recommendation in the comment 
row. 
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ii. Determine if the Sub Activity is appropriate for the Major HSI activity. If 
incorrect, recommend additions, deletions or modifications in the comment 
row. 

b. For Column C, Acquisition Phase: 

i. Determine whether the acquisition phase is correct for the activity. If activity 
occurs during another phase, indicate appropriate phase in comment row 
below activity. 

c. For Column D (HSI Domain {Impacted))'. 

i. Determine if listed domains are affected by HSI activity and sub activity. If 
incorrect, recommend additions or deletions of domains in the comment row. 

d. For Column E (% Effort): 

i. On a 100% scale, estimate to the best of our ability, what percentage of man 
hours is spent on this activity for a generic ACAT I system. Your percentages 
should add to 100% for each phase . (We realize that HSI efforts will vary 
widely depending on the type of system. This column represents an initial 
attempt at determining level of effort required for HSI practitioners.) 

e. For Column F (Cost Drivers): 

i. Review the list of Cost Drivers on Excel file Tab 3 (Cost Drivers). Also 
available on Page 5 of these instructions. 

ii. Determine if listed potential cost drivers are appropriate for activity listed. If 
drivers listed are incorrect, recommend additions or deletions for the activity 
in the comment row. If a cost driver is not listed but should be included, 
please provide the driver in comment row. 

f. For Missing Activities: 

i. If an HSI activity or sub-activity has been left out of the database, please add 
it to the bottom of the activity list. Fill out as much information for the 
columns as possible to ensure the activity it is represented appropriately. 


3. Save the document using Excel (97-2003) compatible file type using 
“samefilename_yourinitials.xls” format. Please send to dasindal@nps.edu no later than 31 
Aug 2010. If more time is needed, please e-mail with request. 

Sources of HSI Data: 

HSI CE activity development was determined through analysis of current instruction, guidance, 
integrated frameworks and past HSI cost studies. Please see Page 6 of these instructions for a 
list of resources. 
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WBS Cost Element 

Numeric Code for organization of Activities based 
on Specific Program Work Breakdown Cost 
Structure (Procurement, O&S, O&M, RDT&E, 
Spares) 

Example Comment: WBS stmcture is generic 
and will be taylored down for specific systems. 

Acquisition Phase 

6 Phases total: Pre-Material Solution Analysis 
(Pre-MSA); Material Solution Analysis (MSA), 
Technology Development Phase (TD); 

Engineering & Manufacturing Development 

Phase: Integrated System Design, System 
Capability and Manufacturing Process 
Demonstration (EMD); Production & Deployment 
(PD); Operation and Support (O&S) 



Figure 43. 


Column Heading Description Example -Q1 (From Review Instructions Page 4) 
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HSI Activity-SME REVIEW - Microsoft Excel 

1 e : 

ii - - 

Home 

Insert 

Page Layout 

Formulas 

Data 

Review 

View 

Add-Ins 

UJ - " 


^ Jt tut 

— 1 Copy 

Paste 

MS Sans Serif - 24 ' A‘ a" 

^ = is?-- 

|i^Wrap Texti 

General 


1- & m 

B I u • H • 1 <3» • ^ ■ 

» « a it it 

^ Merge &. Center ^ 

$ • % * TiS i°8 

Conditional Format Cell 

Formatting’' as TableStyles” 
Styles 

Insert Delete Format 

Clipboard < 

Font ^ 

Alignment ^ 

Number ^ 

Cells 


ifr 


Z AutoSum ’■ 

il Fill - 

Sort & Find & 

^ ClearFilterSelect 




A ^ = 


Editing 



HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

% Effort 

(Man hours) 

Cost_Drivers 

DATA 

Establish a HSI IPT 

Participate in IPT( Incorporate Users, SMEs Engineers, Contractors/ 
Government HSI representatives and other HSI domain experts into 
Integrated Performance Teams). 

Support User-Centric Design Process 

Technology 

Development 

Environment, Habitability, 
Human Factor Engineering, 
Manpower, Occupational 
Health, Personnel, 
Survivability, System Safety, 
Training 


Manning, Personnel experience/continuity 
(Subjective assessment of staff consistency). 
Requirements understanding (Subjective 
assessment of the system requirements). 

Target Audience 

Example 

Input 

This may be part of a bgger activity. 

Need to exapnd *Support User-Centric Design Process 


Ex. Training not involved, 

omit. Ex. 

SHman hours 

required of 10094 

‘Manning* can be used for actual work 
completed (Man hours to perform in IPT) or as a 
driver for system action to be performed on. 

Example 

Input 

All column inputs are acceptable. 







Figure 44. HSI Activity Review Example -Q2 (From Review Instructions Page 3) 
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Table 27. SME Review Potential Column Names and Descriptions for HSI Activity 
Database Development (From Review Instructions, Page 4) 


Column Name 

Descriptbn 

WBS Cost 
Element 

Numeric Code for organization of Activities based on Specific Program Work Breakdown Cost 
Structure (Procurement, O&S, O&M, RDT&E, Spares) 

Acquisition 

Phase 

7 Phases total: Pre-Material Solution Analysis; Material Solution Analysis, Technology 
Development Phase; Engineering & Manufacturing Development Phase; Integrated System 
Design, System Capability and Manufacturing Process Demonstration; Production & Deployment; 
Operation and Support 

Event Category 

Organization of Phase activities based on IDTA&L Defense Interactive Framework (JCIDS Joint 
Capabilities Integration and Development System, Information Technology (IT) & National Security 
(NSS), Earned Value Management (EVM), Oversights Review, Contracting, Major Products, Life 
Cycle Logistics/Sustainment, Technical, Cost 

HSI Activity 

HSI practitioners expected contribution to acquisition process. 

HSI Sub 

Activity 

Detail level orsubset of HSI Activity 

% Effort 

On a scale of 1-100% for each ph ase, the level of HSI practitioner effort contributed in program 
manageme nt. 

For Example: ActivitvOne" Provide HSI activity inputdurinq Capabilities Based Assessment for 

ICD and CDD”, HSI activity is only 3% of man-hour effort for WEAPONS (TRACKED VEH) 
PROCUREMENT (2033 - WTCV). 

HSI Domain 
(Impacted) 

HSI domain affected by the activity being performed, (practitioner, IPT, or domain involvement) 
Environment, Habitability, Human Factor Engineering, Manpower, Occupational Health, Personnel, 
Survivability, System Safety, Training. 

System Type 

Type of system /program that HSI activities will be performed on (i.e . aviation , ship building, 
weapon system, etc). 

Critical Issues 

Operation Effectiveness or Technical Issues that will affect Test and Evaluation Processes (i.e. 
Survivability, Reliability) 

Measures of 
Effectiveness 

Measure designed to correspond to accomplishment of mission objectives and achievement of 
desired results. 

Measure of 
Performance / 
Suitability 

MOP: Measure of a system's performance expressed as speed, payload, range, time on station, 
frequency, or other distinctly quantifiable performance features. MOS: Measure of an item's ability 
to be supported in its intended operational environment. MOSs typically relates to readiness or 
operational availability, and hence neliabiity, maintainability, and the item's support structure. 

Cost Drivers 

An element that impacts the cost of an activity (i.e. Manning, Cognitive Workload, Specialty 
Personnel) '‘Will be used in combination with MOE/MOP to develop potential Cost Element and 

Cost Element Relatbnships for Database Development* 

Benefit 

It defines a solution aimed at achieving specific organizational objectives by quantifying the 
potential financial impacts and other business benefits such as: Savings and/or cost avoidance. 
Revenue enhancements and/or cash-flow improvements. Performance improvements. Are listed 
as quantifiable/non-quantifiable. Used to develop Cost Element Relationships. 

Cost 

Estimation 

Type 

'Program Office Estimate. 'Component Cost Estimate. 'Independent Cost Estimate: * Life-cycle 
cost estimate. 'Total Ownership CE: estimate consists of the elements of life-cycle cost plus some 
infrastructure and business process costs not necessarily attributable to a program. 'Rough Order 
of Magnitu de. 'Independent Cost Assessment, Independent Govern ment Cost Estimate. 

Cost 

Estimation 

Relationships 

complex equations with a number of independent variables. Or Cost Factors (single independent 
variable). Ratbs or percentages use in cost estimation (i.e. parametericcost estimatbn methods). 

Im pact 

Potential Impact of benefit provided. Used to determine Trade-off. Database will include Benefit 
Library with Benefit ID, Description, and Category (guantifiable, non-guantifiable) 

Trade-off 

Potentially weighted domains or benefits that are used for cost over another. Baed on 
prog ram/system 

Comments 

Process, Detail or integration notes relevant to HSI Activity or Sub Activity. 

Target 

Documents 

Source or projected document for which HSI activity occurs (Ex: Contractor Cost Data Reports 

Cost Performance Reports, Selected Acquisition Reports, Recent Defense Acquisition Executive 
Summaries, Unit Cost Reports, Manpower Estimate Reports, HSIP, TEMP, SEP, ITR etc. 

References 

Resources for further database user information. 
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Table 28. SME Review Potential Cost Drivers (from Review Instructions, Page 5) 


Cbgi itive Workload 

Cbmplexity 

QilicalityofTask 

□ffbUtyofTask 

Cbcumertation to match life cyde needs (Bneadthand depthofrequied documentdion) 

Frequency of Ta sk 

hiitialTraining (Instructor & KeyPerscnne 1) 

hiitia 1Train ing (N ew Equ i pm en t a t syste m f iel d ng) 

Level of service requirements (SulqedivedifficulV of satisfying the key performance Parameters) 

IVfointenance Concept 

IVfointenanoe Ftequirerrent 

IVfonnhg 

Mgafon ccmplexity(lrfluenceof legacysistem (if applicabfe)) 

Mssicn 

Mcdelng& Simulatbn 

Mjitisitecocrdination (Locationof stakeholders andcocrdndion harriers) 

Occupatbnal Trahing 

Operational Cepebility 

Ftersonnel s<perience/oortinLity(Subjective assessment of staff consistency) 

Ftersonn el/team capebility (SubjecSveassessment of the team’sinteltecfoal capabilty) 

Physical Workload 

Rdforms(ships, aircraft, weepon systems,etc) 

Fhocess capability (CMMI level or equivdent rding) 

Fte qu ire rre nts untte islanding (Subpcfve assessment of the system req Jrements) 

Sze 

SLOG 

Specidty Personnel 

StakehokJerteam cchesicn (Subjedive assessment ofaIIstekehotiers) 

Target Audie noe 

Technology risk (Nteturity, readiness, an d obsole seen ce of technobg y) 

Time Constraint fa System Development 

Tod support (Sulq'ective assessmentof SE tods) 

lesting lype-detaled component festing 

Testing Type-subsystem integrdion 

Testing Type-system simdationard stimuldion 

Testing Type-campaian or force-level (system ofsvsterrBl. 

Type 

Lhit/Susbinment FiddTranhg 

Warranties 
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